[offers][feat] add script to seed salaries from tech salaries excel sheet (#501)
* [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][chore] Add createdAt and updatedAt to analysis * [offers][chore] Add company name to analysis unit * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix bug where company analysis shows wrong company * [offers][fix] Fix company analysis percentile calculation * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][chore] Change location in schema * Include City, State, and Country in profileDtoMapper params * [offers][fix] Fix merge conflict * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate cityId into create profile endpoint * [offers][feat] integrate location with update profile endpoint * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [resume][refactor] tweak resume submission UI * [offers][chore] Provide more information to frontend in AnalysisUnit * [offers][fix] fix import for salaries script * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][fix] Fix merge conflict * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix import for salaries script * [offers][fix] fix merge conflicts * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][fix] Fix merge conflict * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][feat] Use city typeahead for location field * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix merge conflicts Co-authored-by: Bryann Yeap Kok Keong <bryannyeapkk@gmail.com> Co-authored-by: Ai Ling <hong-ailing@hotmail.com> Co-authored-by: Zhang Ziqing <zhangziqing9926@gmail.com> Co-authored-by: Yangshun Tay <tay.yang.shun@gmail.com>pull/505/head
Stuart Long
2 years ago
committed by
GitHub
parent
32302e33a5
commit
8864a47400
@ -0,0 +1,19 @@
|
||||
import reader from 'xlsx';
|
||||
|
||||
const file = reader.readFile('prisma/salaries.xlsx');
|
||||
|
||||
export const COMPANIES: Array<CompanyData> = []
|
||||
|
||||
type CompanyData = {
|
||||
Finalized: string;
|
||||
description: string;
|
||||
logoUrl: string;
|
||||
name: string;
|
||||
slug: string;
|
||||
website: string;
|
||||
};
|
||||
|
||||
const temp = reader.utils.sheet_to_json(file.Sheets[file.SheetNames[1]]);
|
||||
temp.forEach((res: CompanyData) => {
|
||||
COMPANIES.push(res);
|
||||
});
|
||||
@ -0,0 +1,2 @@
|
||||
-- AlterTable
|
||||
ALTER TABLE "OffersBackground" ALTER COLUMN "totalYoe" SET DEFAULT 0;
|
||||
Binary file not shown.
@ -0,0 +1,379 @@
|
||||
import reader from 'xlsx';
|
||||
import { PrismaClient } from '@prisma/client';
|
||||
import crypto from 'crypto';
|
||||
import { baseCurrencyString } from '../src/utils/offers/currency';
|
||||
import { convert } from '../src/utils/offers/currency/currencyExchange';
|
||||
import { generateAnalysis } from '../src/utils/offers/analysis/analysisGeneration';
|
||||
|
||||
import {
|
||||
generateRandomName,
|
||||
generateRandomStringForToken,
|
||||
} from '../src/utils/offers/randomGenerator';
|
||||
|
||||
const prisma = new PrismaClient();
|
||||
|
||||
// Reading our test file
|
||||
const file = reader.readFile('prisma/salaries.xlsx');
|
||||
|
||||
let data: Array<ExcelData> = [];
|
||||
|
||||
type ExcelData = {
|
||||
Timestamp: Date;
|
||||
Type: string;
|
||||
Company: string;
|
||||
Role: string;
|
||||
Income?: number | string;
|
||||
Stocks?: number | string;
|
||||
SignOn?: number | string;
|
||||
TC?: number | string;
|
||||
Bonus?: number | string;
|
||||
Comments?: string;
|
||||
};
|
||||
|
||||
const sheets = file.SheetNames;
|
||||
|
||||
for (let i = 0; i < 1; i++) {
|
||||
const temp = reader.utils.sheet_to_json(file.Sheets[file.SheetNames[i]]);
|
||||
temp.forEach((res: ExcelData) => {
|
||||
data.push(res);
|
||||
});
|
||||
}
|
||||
|
||||
function xlSerialToJsDate(xlSerial) {
|
||||
return new Date(Date.UTC(0, 0, xlSerial - 1));
|
||||
}
|
||||
|
||||
const getJobTitle = (role: string) => {
|
||||
const processedRole = role.toUpperCase().trim();
|
||||
|
||||
if (processedRole.includes('ML ENGINEER')) {
|
||||
return 'ai-ml-engineer';
|
||||
} else if (processedRole.includes('BACKEND')) {
|
||||
return 'back-end-engineer';
|
||||
} else if (processedRole.includes('DATA')) {
|
||||
return 'data-engineer';
|
||||
} else if (processedRole.includes('DEVOPS')) {
|
||||
return 'devops-engineer';
|
||||
} else if (processedRole.includes('ENTERPRISE')) {
|
||||
return 'enterprise-engineer';
|
||||
} else if (processedRole.includes('RESEARCH')) {
|
||||
return 'research-engineer';
|
||||
} else if (
|
||||
processedRole.includes('CYBER') ||
|
||||
processedRole.includes('SECURITY')
|
||||
) {
|
||||
return 'security-engineer';
|
||||
} else if (processedRole.includes('QA')) {
|
||||
return 'test-engineer';
|
||||
} else if (processedRole.includes('SYSTEM')) {
|
||||
return 'systems-engineer';
|
||||
} else {
|
||||
return 'software-engineer'; // Assume default SWE
|
||||
}
|
||||
};
|
||||
|
||||
const getYoe = (type: string) => {
|
||||
const processedType = type.toUpperCase().trim();
|
||||
|
||||
if (
|
||||
processedType.includes('FRESH GRAD') ||
|
||||
processedType.includes('JUNIOR')
|
||||
) {
|
||||
return Math.floor(Math.random() * 3);
|
||||
} else if (processedType.includes('MID')) {
|
||||
return Math.floor(Math.random() * 3) + 3;
|
||||
} else if (processedType.includes('SENIOR')) {
|
||||
return Math.floor(Math.random() * 5) + 6;
|
||||
} else {
|
||||
return 0; // INTERNSHIP OR ERROR -> 0 YOE
|
||||
}
|
||||
};
|
||||
|
||||
const getLevel = (type: string) => {
|
||||
const processedType = type.toUpperCase().trim();
|
||||
|
||||
if (
|
||||
processedType.includes('FRESH GRAD') ||
|
||||
processedType.includes('JUNIOR')
|
||||
) {
|
||||
return 'L1';
|
||||
} else if (processedType.includes('MID')) {
|
||||
return 'L2';
|
||||
} else if (processedType.includes('SENIOR')) {
|
||||
return 'L4';
|
||||
} else {
|
||||
return 'L0';
|
||||
}
|
||||
};
|
||||
|
||||
const createdProfileIds: Array<string> = [];
|
||||
|
||||
const seedSalaries = async () => {
|
||||
console.log('Seeding from salaries sheet...');
|
||||
|
||||
const companyIdMappings = {};
|
||||
(await prisma.company.findMany()).forEach((company) => {
|
||||
companyIdMappings[company.slug] = company.id;
|
||||
});
|
||||
|
||||
// get countryId of Singapore
|
||||
const singapore = (await prisma.city.findFirst({
|
||||
where: {
|
||||
name: "Singapore"
|
||||
}
|
||||
}))
|
||||
|
||||
console.log("Singapore ID: " + singapore?.id)
|
||||
// break;
|
||||
// seed here
|
||||
|
||||
if (singapore) {
|
||||
return await Promise.all(
|
||||
data.map(async (data: ExcelData) => {
|
||||
if (data.TC && typeof data.TC === 'number') {
|
||||
// Generate random name until unique
|
||||
let uniqueName: string = await generateRandomName();
|
||||
|
||||
const jobTitle = getJobTitle(data.Role);
|
||||
const yoe = getYoe(data.Type);
|
||||
const level = getLevel(data.Type);
|
||||
|
||||
// check if we have company id
|
||||
if (companyIdMappings[data.Company]) {
|
||||
const token = crypto
|
||||
.createHash('sha256')
|
||||
.update(
|
||||
xlSerialToJsDate(data.Timestamp).toString() +
|
||||
generateRandomStringForToken(),
|
||||
)
|
||||
.digest('hex');
|
||||
|
||||
if (data.Type.toUpperCase() === 'INTERNSHIP') {
|
||||
// create profile
|
||||
const dataAdded = await prisma.offersProfile.create({
|
||||
data: {
|
||||
profileName: uniqueName,
|
||||
createdAt: xlSerialToJsDate(data.Timestamp),
|
||||
editToken: token,
|
||||
background: {
|
||||
create: {
|
||||
totalYoe: yoe,
|
||||
},
|
||||
},
|
||||
offers: {
|
||||
create: {
|
||||
comments: data.Comments ?? '',
|
||||
company: {
|
||||
connect: {
|
||||
id: companyIdMappings[data.Company],
|
||||
},
|
||||
},
|
||||
jobType: 'INTERN',
|
||||
location: {
|
||||
connect: {
|
||||
id: singapore.id
|
||||
}
|
||||
}, // TODO: DEFAULT AS SG
|
||||
monthYearReceived: xlSerialToJsDate(data.Timestamp),
|
||||
negotiationStrategy: '',
|
||||
offersIntern: {
|
||||
create: {
|
||||
internshipCycle: 'Summer',
|
||||
monthlySalary: {
|
||||
create: {
|
||||
baseCurrency: baseCurrencyString,
|
||||
baseValue: await convert(
|
||||
data.Income
|
||||
? typeof data.Income === 'number'
|
||||
? data.Income
|
||||
: 0
|
||||
: 0,
|
||||
'SGD', // assume sgd
|
||||
baseCurrencyString,
|
||||
),
|
||||
currency: 'SGD', // assume sgd
|
||||
value: data.Income
|
||||
? typeof data.Income === 'number'
|
||||
? data.Income
|
||||
: 0
|
||||
: 0,
|
||||
},
|
||||
},
|
||||
startYear: xlSerialToJsDate(
|
||||
data.Timestamp,
|
||||
).getFullYear(),
|
||||
title: jobTitle,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
console.log('Profile created:', dataAdded.id);
|
||||
createdProfileIds.push(dataAdded.id);
|
||||
} else {
|
||||
// assume rest full time
|
||||
const dataAdded = await prisma.offersProfile.create({
|
||||
data: {
|
||||
profileName: uniqueName,
|
||||
createdAt: xlSerialToJsDate(data.Timestamp),
|
||||
editToken: token,
|
||||
background: {
|
||||
create: {
|
||||
totalYoe: yoe,
|
||||
},
|
||||
},
|
||||
offers: {
|
||||
create: {
|
||||
comments: data.Comments ?? '',
|
||||
company: {
|
||||
connect: {
|
||||
id: companyIdMappings[data.Company],
|
||||
},
|
||||
},
|
||||
jobType: 'FULLTIME',
|
||||
location: {
|
||||
connect: {
|
||||
id: singapore.id
|
||||
}
|
||||
}, // TODO: DEFAULT AS SG
|
||||
monthYearReceived: xlSerialToJsDate(data.Timestamp),
|
||||
negotiationStrategy: '',
|
||||
offersFullTime: {
|
||||
create: {
|
||||
baseSalary: {
|
||||
create: {
|
||||
baseCurrency: baseCurrencyString,
|
||||
baseValue: await convert(
|
||||
data.Income
|
||||
? typeof data.Income === 'number'
|
||||
? data.Income
|
||||
: 0
|
||||
: 0,
|
||||
'SGD', // assume sgd
|
||||
baseCurrencyString,
|
||||
),
|
||||
currency: 'SGD', // assume sgd
|
||||
value: data.Income
|
||||
? typeof data.Income === 'number'
|
||||
? data.Income
|
||||
: 0
|
||||
: 0,
|
||||
},
|
||||
},
|
||||
bonus: {
|
||||
create: {
|
||||
baseCurrency: baseCurrencyString,
|
||||
baseValue: await convert(
|
||||
data.Bonus
|
||||
? typeof data.Bonus === 'number'
|
||||
? data.Bonus
|
||||
: 0
|
||||
: 0,
|
||||
'SGD',
|
||||
baseCurrencyString,
|
||||
),
|
||||
currency: 'SGD',
|
||||
value: data.Bonus
|
||||
? typeof data.Bonus === 'number'
|
||||
? data.Bonus
|
||||
: 0
|
||||
: 0,
|
||||
},
|
||||
},
|
||||
level: level,
|
||||
stocks: {
|
||||
create: {
|
||||
baseCurrency: baseCurrencyString,
|
||||
baseValue: await convert(
|
||||
data.Stocks
|
||||
? typeof data.Stocks === 'number'
|
||||
? data.Stocks
|
||||
: 0
|
||||
: 0,
|
||||
'SGD',
|
||||
baseCurrencyString,
|
||||
),
|
||||
currency: 'SGD',
|
||||
value: data.Stocks
|
||||
? typeof data.Stocks === 'number'
|
||||
? data.Stocks
|
||||
: 0
|
||||
: 0,
|
||||
},
|
||||
},
|
||||
title: jobTitle,
|
||||
totalCompensation: {
|
||||
create: {
|
||||
baseCurrency: baseCurrencyString,
|
||||
baseValue: await convert(
|
||||
data.TC
|
||||
? typeof data.TC === 'number'
|
||||
? data.TC
|
||||
: 0
|
||||
: 0,
|
||||
'SGD',
|
||||
baseCurrencyString,
|
||||
),
|
||||
currency: 'SGD',
|
||||
value: data.TC
|
||||
? typeof data.TC === 'number'
|
||||
? data.TC
|
||||
: 0
|
||||
: 0,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
console.log('Profile created:', dataAdded.id);
|
||||
createdProfileIds.push(dataAdded.id);
|
||||
}
|
||||
} else {
|
||||
console.log('Invalid Company: ' + data.Company);
|
||||
}
|
||||
} else {
|
||||
console.log('Invalid TC not a number: ' + data.TC);
|
||||
}
|
||||
}),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const generateAllAnalysis = async () => {
|
||||
return await Promise.all(
|
||||
createdProfileIds.map(async (profileId) => {
|
||||
await generateAnalysis({
|
||||
ctx: { prisma, session: null },
|
||||
input: { profileId },
|
||||
});
|
||||
console.log('Analysis generated for profile with id:', profileId);
|
||||
}),
|
||||
);
|
||||
};
|
||||
|
||||
Promise.all([seedSalaries()])
|
||||
.then(() => {
|
||||
console.log(createdProfileIds.length + " profiles created")
|
||||
console.log("Busy crunching analysis.....")
|
||||
})
|
||||
.then(() => generateAllAnalysis())
|
||||
.then((_data) => {
|
||||
console.log('Seeding from salaries sheet complete');
|
||||
})
|
||||
.then(async () => {
|
||||
await prisma.$disconnect();
|
||||
})
|
||||
.catch(async (e) => {
|
||||
console.error(e);
|
||||
await prisma.$disconnect();
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
console.log(xlSerialToJsDate(data[0].Timestamp));
|
||||
|
||||
export {};
|
||||
@ -0,0 +1,33 @@
|
||||
import type { Config } from 'unique-names-generator';
|
||||
import { adjectives, animals,colors, uniqueNamesGenerator } from 'unique-names-generator';
|
||||
import { PrismaClient } from '@prisma/client';
|
||||
|
||||
const prisma = new PrismaClient()
|
||||
|
||||
const customConfig: Config = {
|
||||
dictionaries: [adjectives, colors, animals],
|
||||
length: 3,
|
||||
separator: '-',
|
||||
};
|
||||
|
||||
|
||||
export default async function generateRandomName(): Promise<string> {
|
||||
let uniqueName: string = uniqueNamesGenerator(customConfig);
|
||||
|
||||
let sameNameProfiles = await prisma.offersProfile.findMany({
|
||||
where: {
|
||||
profileName: uniqueName
|
||||
}
|
||||
})
|
||||
|
||||
while (sameNameProfiles.length !== 0) {
|
||||
uniqueName = uniqueNamesGenerator(customConfig);
|
||||
sameNameProfiles = await prisma.offersProfile.findMany({
|
||||
where: {
|
||||
profileName: uniqueName
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
return uniqueName
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in new issue