You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
tech-interview-handbook/apps/portal/prisma/seed-salaries.ts

380 lines
13 KiB

[offers][feat] add script to seed salaries from tech salaries excel sheet (#501) * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][chore] Add createdAt and updatedAt to analysis * [offers][chore] Add company name to analysis unit * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix bug where company analysis shows wrong company * [offers][fix] Fix company analysis percentile calculation * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][chore] Change location in schema * Include City, State, and Country in profileDtoMapper params * [offers][fix] Fix merge conflict * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate cityId into create profile endpoint * [offers][feat] integrate location with update profile endpoint * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [resume][refactor] tweak resume submission UI * [offers][chore] Provide more information to frontend in AnalysisUnit * [offers][fix] fix import for salaries script * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][chore]normalise salaries * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][feat] Add multiple company analysis UI * [offers][fix] Fix empty analysis * [offers][chore] Change user relation in OffersProfile from one-to-many to many-to-many * [offers][fix] Fix merge conflict * [offers][fix] update seeding issue where fulltime base is not year * [offers][fix] update seed script to integrate with city * [offers][chore] Change backend endpoints to new location field * [offers][feat] integrate location for offer table and profile * [offers][chore] fix import of cities typeahead * [offers][feat] Use city typeahead for location field * [offers][fix] fix merge conflict * [offers][fix] fix incorrect salary normalisation * [offers][fix] fix base salary for fulltime * [offers][fix] fix bonus * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix import for salaries script * [offers][fix] fix merge conflicts * [offers][feat] add file to read tech salaries sheet WIP * [offers][chore] read from correct file while seeding * [offers][chore] Add company seed * [offers][feat] add data seeding code to fetch from excel * [offers[chore] Generate analysis for seeded data * [offers][chore] Merge main into branch * [offers][fix] Fix incorrect name * [offers][fix] integrate random name generator to profile seeding * [offers][chore] Set job title in seeding * [offers][chore] Removed specialization * [offers][chore] Add yoe ranges for seeded data * [offers][chore] Rename level params * [offers][fix] add checks for data.income * [offers][chore] Allow analysis to analyse all companies in the backend * [offers][fix] Fix merge conflict * [offers][fix] update seed script to integrate with city * [offers][feat] integrate location for offer table and profile * [offers][feat] Use city typeahead for location field * [offers][chore] add console log to print status while seeding * [offers][feat] normalise sheet to incorporate slug * [offers][feat] read companies from salary sheet * [offers][fix] remove prisma/companySeed.ts from tsconfig.json * [offers][refactor] standardise seed script with question * [offers][fix] fix merge conflicts Co-authored-by: Bryann Yeap Kok Keong <bryannyeapkk@gmail.com> Co-authored-by: Ai Ling <hong-ailing@hotmail.com> Co-authored-by: Zhang Ziqing <zhangziqing9926@gmail.com> Co-authored-by: Yangshun Tay <tay.yang.shun@gmail.com>
2 years ago
import reader from 'xlsx';
import { PrismaClient } from '@prisma/client';
import crypto from 'crypto';
import { baseCurrencyString } from '../src/utils/offers/currency';
import { convert } from '../src/utils/offers/currency/currencyExchange';
import { generateAnalysis } from '../src/utils/offers/analysis/analysisGeneration';
import {
generateRandomName,
generateRandomStringForToken,
} from '../src/utils/offers/randomGenerator';
const prisma = new PrismaClient();
// Reading our test file
const file = reader.readFile('prisma/salaries.xlsx');
let data: Array<ExcelData> = [];
type ExcelData = {
Timestamp: Date;
Type: string;
Company: string;
Role: string;
Income?: number | string;
Stocks?: number | string;
SignOn?: number | string;
TC?: number | string;
Bonus?: number | string;
Comments?: string;
};
const sheets = file.SheetNames;
for (let i = 0; i < 1; i++) {
const temp = reader.utils.sheet_to_json(file.Sheets[file.SheetNames[i]]);
temp.forEach((res: ExcelData) => {
data.push(res);
});
}
function xlSerialToJsDate(xlSerial) {
return new Date(Date.UTC(0, 0, xlSerial - 1));
}
const getJobTitle = (role: string) => {
const processedRole = role.toUpperCase().trim();
if (processedRole.includes('ML ENGINEER')) {
return 'ai-ml-engineer';
} else if (processedRole.includes('BACKEND')) {
return 'back-end-engineer';
} else if (processedRole.includes('DATA')) {
return 'data-engineer';
} else if (processedRole.includes('DEVOPS')) {
return 'devops-engineer';
} else if (processedRole.includes('ENTERPRISE')) {
return 'enterprise-engineer';
} else if (processedRole.includes('RESEARCH')) {
return 'research-engineer';
} else if (
processedRole.includes('CYBER') ||
processedRole.includes('SECURITY')
) {
return 'security-engineer';
} else if (processedRole.includes('QA')) {
return 'test-engineer';
} else if (processedRole.includes('SYSTEM')) {
return 'systems-engineer';
} else {
return 'software-engineer'; // Assume default SWE
}
};
const getYoe = (type: string) => {
const processedType = type.toUpperCase().trim();
if (
processedType.includes('FRESH GRAD') ||
processedType.includes('JUNIOR')
) {
return Math.floor(Math.random() * 3);
} else if (processedType.includes('MID')) {
return Math.floor(Math.random() * 3) + 3;
} else if (processedType.includes('SENIOR')) {
return Math.floor(Math.random() * 5) + 6;
} else {
return 0; // INTERNSHIP OR ERROR -> 0 YOE
}
};
const getLevel = (type: string) => {
const processedType = type.toUpperCase().trim();
if (
processedType.includes('FRESH GRAD') ||
processedType.includes('JUNIOR')
) {
return 'L1';
} else if (processedType.includes('MID')) {
return 'L2';
} else if (processedType.includes('SENIOR')) {
return 'L4';
} else {
return 'L0';
}
};
const createdProfileIds: Array<string> = [];
const seedSalaries = async () => {
console.log('Seeding from salaries sheet...');
const companyIdMappings = {};
(await prisma.company.findMany()).forEach((company) => {
companyIdMappings[company.slug] = company.id;
});
// get countryId of Singapore
const singapore = (await prisma.city.findFirst({
where: {
name: "Singapore"
}
}))
console.log("Singapore ID: " + singapore?.id)
// break;
// seed here
if (singapore) {
return await Promise.all(
data.map(async (data: ExcelData) => {
if (data.TC && typeof data.TC === 'number') {
// Generate random name until unique
let uniqueName: string = await generateRandomName();
const jobTitle = getJobTitle(data.Role);
const yoe = getYoe(data.Type);
const level = getLevel(data.Type);
// check if we have company id
if (companyIdMappings[data.Company]) {
const token = crypto
.createHash('sha256')
.update(
xlSerialToJsDate(data.Timestamp).toString() +
generateRandomStringForToken(),
)
.digest('hex');
if (data.Type.toUpperCase() === 'INTERNSHIP') {
// create profile
const dataAdded = await prisma.offersProfile.create({
data: {
profileName: uniqueName,
createdAt: xlSerialToJsDate(data.Timestamp),
editToken: token,
background: {
create: {
totalYoe: yoe,
},
},
offers: {
create: {
comments: data.Comments ?? '',
company: {
connect: {
id: companyIdMappings[data.Company],
},
},
jobType: 'INTERN',
location: {
connect: {
id: singapore.id
}
}, // TODO: DEFAULT AS SG
monthYearReceived: xlSerialToJsDate(data.Timestamp),
negotiationStrategy: '',
offersIntern: {
create: {
internshipCycle: 'Summer',
monthlySalary: {
create: {
baseCurrency: baseCurrencyString,
baseValue: await convert(
data.Income
? typeof data.Income === 'number'
? data.Income
: 0
: 0,
'SGD', // assume sgd
baseCurrencyString,
),
currency: 'SGD', // assume sgd
value: data.Income
? typeof data.Income === 'number'
? data.Income
: 0
: 0,
},
},
startYear: xlSerialToJsDate(
data.Timestamp,
).getFullYear(),
title: jobTitle,
},
},
},
},
},
});
console.log('Profile created:', dataAdded.id);
createdProfileIds.push(dataAdded.id);
} else {
// assume rest full time
const dataAdded = await prisma.offersProfile.create({
data: {
profileName: uniqueName,
createdAt: xlSerialToJsDate(data.Timestamp),
editToken: token,
background: {
create: {
totalYoe: yoe,
},
},
offers: {
create: {
comments: data.Comments ?? '',
company: {
connect: {
id: companyIdMappings[data.Company],
},
},
jobType: 'FULLTIME',
location: {
connect: {
id: singapore.id
}
}, // TODO: DEFAULT AS SG
monthYearReceived: xlSerialToJsDate(data.Timestamp),
negotiationStrategy: '',
offersFullTime: {
create: {
baseSalary: {
create: {
baseCurrency: baseCurrencyString,
baseValue: await convert(
data.Income
? typeof data.Income === 'number'
? data.Income
: 0
: 0,
'SGD', // assume sgd
baseCurrencyString,
),
currency: 'SGD', // assume sgd
value: data.Income
? typeof data.Income === 'number'
? data.Income
: 0
: 0,
},
},
bonus: {
create: {
baseCurrency: baseCurrencyString,
baseValue: await convert(
data.Bonus
? typeof data.Bonus === 'number'
? data.Bonus
: 0
: 0,
'SGD',
baseCurrencyString,
),
currency: 'SGD',
value: data.Bonus
? typeof data.Bonus === 'number'
? data.Bonus
: 0
: 0,
},
},
level: level,
stocks: {
create: {
baseCurrency: baseCurrencyString,
baseValue: await convert(
data.Stocks
? typeof data.Stocks === 'number'
? data.Stocks
: 0
: 0,
'SGD',
baseCurrencyString,
),
currency: 'SGD',
value: data.Stocks
? typeof data.Stocks === 'number'
? data.Stocks
: 0
: 0,
},
},
title: jobTitle,
totalCompensation: {
create: {
baseCurrency: baseCurrencyString,
baseValue: await convert(
data.TC
? typeof data.TC === 'number'
? data.TC
: 0
: 0,
'SGD',
baseCurrencyString,
),
currency: 'SGD',
value: data.TC
? typeof data.TC === 'number'
? data.TC
: 0
: 0,
},
},
},
},
},
},
},
});
console.log('Profile created:', dataAdded.id);
createdProfileIds.push(dataAdded.id);
}
} else {
console.log('Invalid Company: ' + data.Company);
}
} else {
console.log('Invalid TC not a number: ' + data.TC);
}
}),
);
}
};
const generateAllAnalysis = async () => {
return await Promise.all(
createdProfileIds.map(async (profileId) => {
await generateAnalysis({
ctx: { prisma, session: null },
input: { profileId },
});
console.log('Analysis generated for profile with id:', profileId);
}),
);
};
Promise.all([seedSalaries()])
.then(() => {
console.log(createdProfileIds.length + " profiles created")
console.log("Busy crunching analysis.....")
})
.then(() => generateAllAnalysis())
.then((_data) => {
console.log('Seeding from salaries sheet complete');
})
.then(async () => {
await prisma.$disconnect();
})
.catch(async (e) => {
console.error(e);
await prisma.$disconnect();
process.exit(1);
});
console.log(xlSerialToJsDate(data[0].Timestamp));
export {};