[offers][feat] add script to seed salaries from tech salaries excel sheet (#501)
* [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][chore] Add createdAt and updatedAt to analysis * [offers][chore] Add company name to analysis unit * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix bug where company analysis shows wrong company * [offers][fix] Fix company analysis percentile calculation * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][chore] Change location in schema * Include City, State, and Country in profileDtoMapper params * [offers][fix] Fix merge conflict * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate cityId into create profile endpoint * [offers][feat] integrate location with update profile endpoint * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [resume][refactor] tweak resume submission UI * [offers][chore] Provide more information to frontend in AnalysisUnit * [offers][fix] fix import for salaries script * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][fix] Fix merge conflict * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix import for salaries script * [offers][fix] fix merge conflicts * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][fix] Fix merge conflict * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][feat] Use city typeahead for location field * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix merge conflicts Co-authored-by: Bryann Yeap Kok Keong <bryannyeapkk@gmail.com> Co-authored-by: Ai Ling <hong-ailing@hotmail.com> Co-authored-by: Zhang Ziqing <zhangziqing9926@gmail.com> Co-authored-by: Yangshun Tay <tay.yang.shun@gmail.com>pull/505/head
Stuart Long
2 years ago
committed by
GitHub
parent
32302e33a5
commit
8864a47400
@ -0,0 +1,19 @@
|
|||||||
|
import reader from 'xlsx';
|
||||||
|
|
||||||
|
const file = reader.readFile('prisma/salaries.xlsx');
|
||||||
|
|
||||||
|
export const COMPANIES: Array<CompanyData> = []
|
||||||
|
|
||||||
|
type CompanyData = {
|
||||||
|
Finalized: string;
|
||||||
|
description: string;
|
||||||
|
logoUrl: string;
|
||||||
|
name: string;
|
||||||
|
slug: string;
|
||||||
|
website: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const temp = reader.utils.sheet_to_json(file.Sheets[file.SheetNames[1]]);
|
||||||
|
temp.forEach((res: CompanyData) => {
|
||||||
|
COMPANIES.push(res);
|
||||||
|
});
|
||||||
@ -0,0 +1,2 @@
|
|||||||
|
-- AlterTable
|
||||||
|
ALTER TABLE "OffersBackground" ALTER COLUMN "totalYoe" SET DEFAULT 0;
|
||||||
Binary file not shown.
@ -0,0 +1,379 @@
|
|||||||
|
import reader from 'xlsx';
|
||||||
|
import { PrismaClient } from '@prisma/client';
|
||||||
|
import crypto from 'crypto';
|
||||||
|
import { baseCurrencyString } from '../src/utils/offers/currency';
|
||||||
|
import { convert } from '../src/utils/offers/currency/currencyExchange';
|
||||||
|
import { generateAnalysis } from '../src/utils/offers/analysis/analysisGeneration';
|
||||||
|
|
||||||
|
import {
|
||||||
|
generateRandomName,
|
||||||
|
generateRandomStringForToken,
|
||||||
|
} from '../src/utils/offers/randomGenerator';
|
||||||
|
|
||||||
|
const prisma = new PrismaClient();
|
||||||
|
|
||||||
|
// Reading our test file
|
||||||
|
const file = reader.readFile('prisma/salaries.xlsx');
|
||||||
|
|
||||||
|
let data: Array<ExcelData> = [];
|
||||||
|
|
||||||
|
type ExcelData = {
|
||||||
|
Timestamp: Date;
|
||||||
|
Type: string;
|
||||||
|
Company: string;
|
||||||
|
Role: string;
|
||||||
|
Income?: number | string;
|
||||||
|
Stocks?: number | string;
|
||||||
|
SignOn?: number | string;
|
||||||
|
TC?: number | string;
|
||||||
|
Bonus?: number | string;
|
||||||
|
Comments?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const sheets = file.SheetNames;
|
||||||
|
|
||||||
|
for (let i = 0; i < 1; i++) {
|
||||||
|
const temp = reader.utils.sheet_to_json(file.Sheets[file.SheetNames[i]]);
|
||||||
|
temp.forEach((res: ExcelData) => {
|
||||||
|
data.push(res);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function xlSerialToJsDate(xlSerial) {
|
||||||
|
return new Date(Date.UTC(0, 0, xlSerial - 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
const getJobTitle = (role: string) => {
|
||||||
|
const processedRole = role.toUpperCase().trim();
|
||||||
|
|
||||||
|
if (processedRole.includes('ML ENGINEER')) {
|
||||||
|
return 'ai-ml-engineer';
|
||||||
|
} else if (processedRole.includes('BACKEND')) {
|
||||||
|
return 'back-end-engineer';
|
||||||
|
} else if (processedRole.includes('DATA')) {
|
||||||
|
return 'data-engineer';
|
||||||
|
} else if (processedRole.includes('DEVOPS')) {
|
||||||
|
return 'devops-engineer';
|
||||||
|
} else if (processedRole.includes('ENTERPRISE')) {
|
||||||
|
return 'enterprise-engineer';
|
||||||
|
} else if (processedRole.includes('RESEARCH')) {
|
||||||
|
return 'research-engineer';
|
||||||
|
} else if (
|
||||||
|
processedRole.includes('CYBER') ||
|
||||||
|
processedRole.includes('SECURITY')
|
||||||
|
) {
|
||||||
|
return 'security-engineer';
|
||||||
|
} else if (processedRole.includes('QA')) {
|
||||||
|
return 'test-engineer';
|
||||||
|
} else if (processedRole.includes('SYSTEM')) {
|
||||||
|
return 'systems-engineer';
|
||||||
|
} else {
|
||||||
|
return 'software-engineer'; // Assume default SWE
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const getYoe = (type: string) => {
|
||||||
|
const processedType = type.toUpperCase().trim();
|
||||||
|
|
||||||
|
if (
|
||||||
|
processedType.includes('FRESH GRAD') ||
|
||||||
|
processedType.includes('JUNIOR')
|
||||||
|
) {
|
||||||
|
return Math.floor(Math.random() * 3);
|
||||||
|
} else if (processedType.includes('MID')) {
|
||||||
|
return Math.floor(Math.random() * 3) + 3;
|
||||||
|
} else if (processedType.includes('SENIOR')) {
|
||||||
|
return Math.floor(Math.random() * 5) + 6;
|
||||||
|
} else {
|
||||||
|
return 0; // INTERNSHIP OR ERROR -> 0 YOE
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const getLevel = (type: string) => {
|
||||||
|
const processedType = type.toUpperCase().trim();
|
||||||
|
|
||||||
|
if (
|
||||||
|
processedType.includes('FRESH GRAD') ||
|
||||||
|
processedType.includes('JUNIOR')
|
||||||
|
) {
|
||||||
|
return 'L1';
|
||||||
|
} else if (processedType.includes('MID')) {
|
||||||
|
return 'L2';
|
||||||
|
} else if (processedType.includes('SENIOR')) {
|
||||||
|
return 'L4';
|
||||||
|
} else {
|
||||||
|
return 'L0';
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const createdProfileIds: Array<string> = [];
|
||||||
|
|
||||||
|
const seedSalaries = async () => {
|
||||||
|
console.log('Seeding from salaries sheet...');
|
||||||
|
|
||||||
|
const companyIdMappings = {};
|
||||||
|
(await prisma.company.findMany()).forEach((company) => {
|
||||||
|
companyIdMappings[company.slug] = company.id;
|
||||||
|
});
|
||||||
|
|
||||||
|
// get countryId of Singapore
|
||||||
|
const singapore = (await prisma.city.findFirst({
|
||||||
|
where: {
|
||||||
|
name: "Singapore"
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
|
||||||
|
console.log("Singapore ID: " + singapore?.id)
|
||||||
|
// break;
|
||||||
|
// seed here
|
||||||
|
|
||||||
|
if (singapore) {
|
||||||
|
return await Promise.all(
|
||||||
|
data.map(async (data: ExcelData) => {
|
||||||
|
if (data.TC && typeof data.TC === 'number') {
|
||||||
|
// Generate random name until unique
|
||||||
|
let uniqueName: string = await generateRandomName();
|
||||||
|
|
||||||
|
const jobTitle = getJobTitle(data.Role);
|
||||||
|
const yoe = getYoe(data.Type);
|
||||||
|
const level = getLevel(data.Type);
|
||||||
|
|
||||||
|
// check if we have company id
|
||||||
|
if (companyIdMappings[data.Company]) {
|
||||||
|
const token = crypto
|
||||||
|
.createHash('sha256')
|
||||||
|
.update(
|
||||||
|
xlSerialToJsDate(data.Timestamp).toString() +
|
||||||
|
generateRandomStringForToken(),
|
||||||
|
)
|
||||||
|
.digest('hex');
|
||||||
|
|
||||||
|
if (data.Type.toUpperCase() === 'INTERNSHIP') {
|
||||||
|
// create profile
|
||||||
|
const dataAdded = await prisma.offersProfile.create({
|
||||||
|
data: {
|
||||||
|
profileName: uniqueName,
|
||||||
|
createdAt: xlSerialToJsDate(data.Timestamp),
|
||||||
|
editToken: token,
|
||||||
|
background: {
|
||||||
|
create: {
|
||||||
|
totalYoe: yoe,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
offers: {
|
||||||
|
create: {
|
||||||
|
comments: data.Comments ?? '',
|
||||||
|
company: {
|
||||||
|
connect: {
|
||||||
|
id: companyIdMappings[data.Company],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
jobType: 'INTERN',
|
||||||
|
location: {
|
||||||
|
connect: {
|
||||||
|
id: singapore.id
|
||||||
|
}
|
||||||
|
}, // TODO: DEFAULT AS SG
|
||||||
|
monthYearReceived: xlSerialToJsDate(data.Timestamp),
|
||||||
|
negotiationStrategy: '',
|
||||||
|
offersIntern: {
|
||||||
|
create: {
|
||||||
|
internshipCycle: 'Summer',
|
||||||
|
monthlySalary: {
|
||||||
|
create: {
|
||||||
|
baseCurrency: baseCurrencyString,
|
||||||
|
baseValue: await convert(
|
||||||
|
data.Income
|
||||||
|
? typeof data.Income === 'number'
|
||||||
|
? data.Income
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
'SGD', // assume sgd
|
||||||
|
baseCurrencyString,
|
||||||
|
),
|
||||||
|
currency: 'SGD', // assume sgd
|
||||||
|
value: data.Income
|
||||||
|
? typeof data.Income === 'number'
|
||||||
|
? data.Income
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
startYear: xlSerialToJsDate(
|
||||||
|
data.Timestamp,
|
||||||
|
).getFullYear(),
|
||||||
|
title: jobTitle,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log('Profile created:', dataAdded.id);
|
||||||
|
createdProfileIds.push(dataAdded.id);
|
||||||
|
} else {
|
||||||
|
// assume rest full time
|
||||||
|
const dataAdded = await prisma.offersProfile.create({
|
||||||
|
data: {
|
||||||
|
profileName: uniqueName,
|
||||||
|
createdAt: xlSerialToJsDate(data.Timestamp),
|
||||||
|
editToken: token,
|
||||||
|
background: {
|
||||||
|
create: {
|
||||||
|
totalYoe: yoe,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
offers: {
|
||||||
|
create: {
|
||||||
|
comments: data.Comments ?? '',
|
||||||
|
company: {
|
||||||
|
connect: {
|
||||||
|
id: companyIdMappings[data.Company],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
jobType: 'FULLTIME',
|
||||||
|
location: {
|
||||||
|
connect: {
|
||||||
|
id: singapore.id
|
||||||
|
}
|
||||||
|
}, // TODO: DEFAULT AS SG
|
||||||
|
monthYearReceived: xlSerialToJsDate(data.Timestamp),
|
||||||
|
negotiationStrategy: '',
|
||||||
|
offersFullTime: {
|
||||||
|
create: {
|
||||||
|
baseSalary: {
|
||||||
|
create: {
|
||||||
|
baseCurrency: baseCurrencyString,
|
||||||
|
baseValue: await convert(
|
||||||
|
data.Income
|
||||||
|
? typeof data.Income === 'number'
|
||||||
|
? data.Income
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
'SGD', // assume sgd
|
||||||
|
baseCurrencyString,
|
||||||
|
),
|
||||||
|
currency: 'SGD', // assume sgd
|
||||||
|
value: data.Income
|
||||||
|
? typeof data.Income === 'number'
|
||||||
|
? data.Income
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
bonus: {
|
||||||
|
create: {
|
||||||
|
baseCurrency: baseCurrencyString,
|
||||||
|
baseValue: await convert(
|
||||||
|
data.Bonus
|
||||||
|
? typeof data.Bonus === 'number'
|
||||||
|
? data.Bonus
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
'SGD',
|
||||||
|
baseCurrencyString,
|
||||||
|
),
|
||||||
|
currency: 'SGD',
|
||||||
|
value: data.Bonus
|
||||||
|
? typeof data.Bonus === 'number'
|
||||||
|
? data.Bonus
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
level: level,
|
||||||
|
stocks: {
|
||||||
|
create: {
|
||||||
|
baseCurrency: baseCurrencyString,
|
||||||
|
baseValue: await convert(
|
||||||
|
data.Stocks
|
||||||
|
? typeof data.Stocks === 'number'
|
||||||
|
? data.Stocks
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
'SGD',
|
||||||
|
baseCurrencyString,
|
||||||
|
),
|
||||||
|
currency: 'SGD',
|
||||||
|
value: data.Stocks
|
||||||
|
? typeof data.Stocks === 'number'
|
||||||
|
? data.Stocks
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
title: jobTitle,
|
||||||
|
totalCompensation: {
|
||||||
|
create: {
|
||||||
|
baseCurrency: baseCurrencyString,
|
||||||
|
baseValue: await convert(
|
||||||
|
data.TC
|
||||||
|
? typeof data.TC === 'number'
|
||||||
|
? data.TC
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
'SGD',
|
||||||
|
baseCurrencyString,
|
||||||
|
),
|
||||||
|
currency: 'SGD',
|
||||||
|
value: data.TC
|
||||||
|
? typeof data.TC === 'number'
|
||||||
|
? data.TC
|
||||||
|
: 0
|
||||||
|
: 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
console.log('Profile created:', dataAdded.id);
|
||||||
|
createdProfileIds.push(dataAdded.id);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log('Invalid Company: ' + data.Company);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log('Invalid TC not a number: ' + data.TC);
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const generateAllAnalysis = async () => {
|
||||||
|
return await Promise.all(
|
||||||
|
createdProfileIds.map(async (profileId) => {
|
||||||
|
await generateAnalysis({
|
||||||
|
ctx: { prisma, session: null },
|
||||||
|
input: { profileId },
|
||||||
|
});
|
||||||
|
console.log('Analysis generated for profile with id:', profileId);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
Promise.all([seedSalaries()])
|
||||||
|
.then(() => {
|
||||||
|
console.log(createdProfileIds.length + " profiles created")
|
||||||
|
console.log("Busy crunching analysis.....")
|
||||||
|
})
|
||||||
|
.then(() => generateAllAnalysis())
|
||||||
|
.then((_data) => {
|
||||||
|
console.log('Seeding from salaries sheet complete');
|
||||||
|
})
|
||||||
|
.then(async () => {
|
||||||
|
await prisma.$disconnect();
|
||||||
|
})
|
||||||
|
.catch(async (e) => {
|
||||||
|
console.error(e);
|
||||||
|
await prisma.$disconnect();
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log(xlSerialToJsDate(data[0].Timestamp));
|
||||||
|
|
||||||
|
export {};
|
||||||
@ -0,0 +1,33 @@
|
|||||||
|
import type { Config } from 'unique-names-generator';
|
||||||
|
import { adjectives, animals,colors, uniqueNamesGenerator } from 'unique-names-generator';
|
||||||
|
import { PrismaClient } from '@prisma/client';
|
||||||
|
|
||||||
|
const prisma = new PrismaClient()
|
||||||
|
|
||||||
|
const customConfig: Config = {
|
||||||
|
dictionaries: [adjectives, colors, animals],
|
||||||
|
length: 3,
|
||||||
|
separator: '-',
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
export default async function generateRandomName(): Promise<string> {
|
||||||
|
let uniqueName: string = uniqueNamesGenerator(customConfig);
|
||||||
|
|
||||||
|
let sameNameProfiles = await prisma.offersProfile.findMany({
|
||||||
|
where: {
|
||||||
|
profileName: uniqueName
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
while (sameNameProfiles.length !== 0) {
|
||||||
|
uniqueName = uniqueNamesGenerator(customConfig);
|
||||||
|
sameNameProfiles = await prisma.offersProfile.findMany({
|
||||||
|
where: {
|
||||||
|
profileName: uniqueName
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return uniqueName
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in new issue