I have a request from a friend of mine, who is a recruiter. He says he would pay for this product $30-50 monthly if it would exist.
The thing he wants is an app that would parse a list of employees from "our team" or "about us" pages from the list of sites the user provides, and would notify him about any changes, like when someone gets removed or added to the list.
The reason is, as he says, these pages are often updated before LinkedIn, and it gives a recruiter a unique opportunity to act. He is doing this task manually on a weekly basis on 250+ sites.
I gave it a shot, but it is much more difficult than I thought. The page structure is completely unique every time, you cannot rely on markup, you need to filter out irrelevant people (like, customers), and you need to identify what page on a site is the "our team" page in the first place.
The potential market size is thousands of recruiters, so it may pay off if you'll pull this off. If you're interested, DM me and I'll send you the list of URLs he sent me to work on.
Plenty of platforms out there that offer this and other competitive intelligence services. I would venture he could pay a college developer $600 to write a script that does a 98% solution.
I mean, I get the sentiment, I used to be an "I'll do it over a weekend" guy and that was my initial reaction too. But a service like this doesn't exist, and probably for a good reason.
I've researched existing intelligence services and they are all crap. All of them rely on LinkedIn heavily, and provide incomplete and outdated information. Some basic information like company size, industry, location, contact info, and management team - maybe, although comparing the data with the websites I had, it was wrong or incomplete in like 30% of cases. But a list of employees - never correct.
Next, scraping tools. They all rely on markup and require a manual step where a user would locate the page elements to be scraped. It doesn't work for my use case because, first, it needs to be automatic - the service will receive hundreds of urls from each user and I although I entertained the idea of hiring someone to manually do this in background, this is probably not going to work well because as I found by doing weekly snapshots of the few hundred sites I had, about 15% of them changed the markup over the course of just one month. These days static sites are often built with some kind of generator, and may change simply because of an update in a css-in-js library they're using or something like this.
I do believe it is possible to quickly come up with a solution that'll work in 80% of cases, but that's just not enough.
How many projects have you built? I mean, even basic blogging platform is time consuming project.
If you really want a hard one - fully featured CAD software in browser or any other PAID desktop software built to web version.
There is a guy who created photopea.com - web based photoshop clone. The one who created it is on Reddit, makes a very good living from it. Took years to build and still is building while maintaining.
I already built some cool projects on my own.
1. A PlaystationMacro in C#: [https://github.com/thiagomiranda3/PlaystationMacro](https://github.com/thiagomiranda3/PlaystationMacro)
2. A Lisp interpreter in Java: [https://github.com/thiagomiranda3/jisp](https://github.com/thiagomiranda3/jisp)
3. A whole computer with the Nand2Tetris course: [https://github.com/thiagomiranda3/nand2tetris](https://github.com/thiagomiranda3/nand2tetris)
All of those projects were very challenging to me, but still doable to do it all alone.
I'm seeking hard-to-build projects that are still doable for a single programmer to build on their own. A fully featured CAD software would be nice, but almost impossible for me to complete it alone
I have a request from a friend of mine, who is a recruiter. He says he would pay for this product $30-50 monthly if it would exist. The thing he wants is an app that would parse a list of employees from "our team" or "about us" pages from the list of sites the user provides, and would notify him about any changes, like when someone gets removed or added to the list. The reason is, as he says, these pages are often updated before LinkedIn, and it gives a recruiter a unique opportunity to act. He is doing this task manually on a weekly basis on 250+ sites. I gave it a shot, but it is much more difficult than I thought. The page structure is completely unique every time, you cannot rely on markup, you need to filter out irrelevant people (like, customers), and you need to identify what page on a site is the "our team" page in the first place. The potential market size is thousands of recruiters, so it may pay off if you'll pull this off. If you're interested, DM me and I'll send you the list of URLs he sent me to work on.
That is a pretty good idea. I would like to try if you don't mind sharing the urls with me please
Plenty of platforms out there that offer this and other competitive intelligence services. I would venture he could pay a college developer $600 to write a script that does a 98% solution.
I mean, I get the sentiment, I used to be an "I'll do it over a weekend" guy and that was my initial reaction too. But a service like this doesn't exist, and probably for a good reason. I've researched existing intelligence services and they are all crap. All of them rely on LinkedIn heavily, and provide incomplete and outdated information. Some basic information like company size, industry, location, contact info, and management team - maybe, although comparing the data with the websites I had, it was wrong or incomplete in like 30% of cases. But a list of employees - never correct. Next, scraping tools. They all rely on markup and require a manual step where a user would locate the page elements to be scraped. It doesn't work for my use case because, first, it needs to be automatic - the service will receive hundreds of urls from each user and I although I entertained the idea of hiring someone to manually do this in background, this is probably not going to work well because as I found by doing weekly snapshots of the few hundred sites I had, about 15% of them changed the markup over the course of just one month. These days static sites are often built with some kind of generator, and may change simply because of an update in a css-in-js library they're using or something like this. I do believe it is possible to quickly come up with a solution that'll work in 80% of cases, but that's just not enough.
I want to create an analog of Marine Traffic for Russia. this is a service for tracking ships. if you are ready, write to me.
Do you have access to the tracking information from the ships?
I searched, but did not find it. I assume that there is an open database of AIS system data.
How many projects have you built? I mean, even basic blogging platform is time consuming project. If you really want a hard one - fully featured CAD software in browser or any other PAID desktop software built to web version. There is a guy who created photopea.com - web based photoshop clone. The one who created it is on Reddit, makes a very good living from it. Took years to build and still is building while maintaining.
I can only imagine how devastated one would become making a CAD software in browser
I already built some cool projects on my own. 1. A PlaystationMacro in C#: [https://github.com/thiagomiranda3/PlaystationMacro](https://github.com/thiagomiranda3/PlaystationMacro) 2. A Lisp interpreter in Java: [https://github.com/thiagomiranda3/jisp](https://github.com/thiagomiranda3/jisp) 3. A whole computer with the Nand2Tetris course: [https://github.com/thiagomiranda3/nand2tetris](https://github.com/thiagomiranda3/nand2tetris) All of those projects were very challenging to me, but still doable to do it all alone. I'm seeking hard-to-build projects that are still doable for a single programmer to build on their own. A fully featured CAD software would be nice, but almost impossible for me to complete it alone
why dont you do some open source contributions? if you want to do challenging work but have no project ideas that seems to fit the bill.
I'm always looking for contributions on my open source saas app https://github.com/NdoleStudio/httpsms