Skip to main content

Raymii.org Raymii.org Logo

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed

Hacker News vs Lobste.rs in C++, an exercise in parsing json http api's and date/time/timezones

Published: 31-12-2020 | Author: Remy van Elst | Text only version of this article


❗ This post is over three years old. It may no longer be up to date. Opinions may have changed.

screenshot

I recently wondered how many top posts on the Hacker News frontpage are also on Lobsters. At first I reached for my trusty Python, because when I need to do some JSON API parsing that's what I'll use. (Otherwise bash is my default goto for small things, except when json, networking or associative arrays are involved.)

But, then, a thought came to my mind. Why not try it with reasonably modern C++. It's what I do at work, so why not a simple personal project. It would involve dependency management (json & http library), parsing both API endpoints and, most importantly, doing stuff with time. Time, timezones and dates are hard.

This article contains a bit of my learning process, compilation and usage instructions and an example run. Go look at the code and run the code yourself. Let me know if my timezone calculations are working outside of GMT+1.

All of the above things I can do and setup in less than an hour with Python and Requests, even urllib if I have to. But C++ is a different story. No built in http client, no JSON parser, dependency management isn't easy (no boost allowed this time), which is why I went for header only libraries (that's by far the easiest solution).

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!

Using std::async makes firing off multiple http requests a breeze. There is even a sort of queue. The code is simple enough to serve as an example for studying.

Lobsters has the post timestamp in CST and Hackernews has just the epoch. My local timezone is GMT+1 and the date/time calculations are the most important part of this script. Modern C++ has little going for itself on parsing time/date, some parts are in std::chrono but most of it is in <ctime>. Or boost if you fancy that. When do you use gmtime over localtime, a time_t vs a struct tm, all that fun stuff. I knew the theory, but doing it in practice helps a tremendous way.

What did I learn with this endeavor:

  • HTTP / HTTPS request with cpp-httplib
  • JSON parsing with nlohmann::json
  • Parsing a string date/time in a different timezone
  • When to use timegm over mktime, as for localtime vs gmtime.
  • Parallel http requests via std::async

I used to cross post, but nowadays only post to Lobsters, since todsaceroti posts my stuff to Hackernews as soon as the RSS feed lists it. I suspect that being a bot account. Not complaining BTW, self-submission is frowned upon in some circles, so it's nice to see someone else posting my stuff.

Get the source code on Github.

I'm planning a few follow up articles on the specific topics, like the date/time parsing or std::async.

Installation (Compilation)

Usual cmake steps, dependency on OpenSSL (apt install libssl-dev)

Clone the repository:

git clone https://github.com/RaymiiOrg/lobsters-hn-post-compare
cd lobsters-hn-post-compare

Setup a cmake folder:

mkdir build
cd build
cmake ..

Compile:

make

Binary hn_lob_comp should be in the same folder.

Usage

chmod +x ./hn_lob_comp
./hn_lob_comp help

Output:

Which stories appear both on Lobsters and on HN, who was first?
An excuse to play with parsing a JSON api in C++ with async by Remy van Elst (https://raymii.org)

Current date/time: 2020-12-30T22:21:43 +0100

Usage: ./hn_lob_comp [help|test|top|new]
./hn_lob_comp top: analyze top stories from HN & Lobsters.
./hn_lob_comp help: this text.
./hn_lob_comp test: run a test to check your timezones.
./hn_lob_comp new: get new posts instead of best.

You'll probably want the top command:

./hn_lob_comp top

Output

Here's what a top run looks like:

Which stories appear both on Lobsters and on HN, who was first?
An excuse to play with parsing a JSON api in C++ with async by Remy van Elst (https://raymii.org)

Current date/time: 2020-12-30T22:24:57 +0100

Fetching HackerNews Best Stories async (200 posts) (https://github.com/HackerNews/API)
Fetching the first ten Lobsters pages async 10*25=200 posts) (https://lobste.rs/s/r9oskz/is_there_api_documentation_for_lobsters_somewhere)

Number of posts from Lobsters    : 200
Number of posts from Hacker News : 192

Matches (13):

# ACE: Apple Type-C Port Controller Secrets  
URL: https://blog.t8012.dev/ace-part-1/  
First appeared on **HackerNews** with 257 votes and 106 comments, submitted by aunali1 (2020-12-30T07:49:16 +0100; https://news.ycombinator.com/item?id=25579286 ).  
After 6 hours, 19 minutes, 28 seconds it was submitted to **Lobsters** by calvin with 9 votes and 1 comments (2020-12-30T14:08:44 +0100; https://lobste.rs/s/so3rb4/ace_apple_type_c_port_controller_secrets ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Computer Science textbooks that are freely available online  
URL: https://csgordon.github.io/books.html  
First appeared on **HackerNews** with 534 votes and 50 comments, submitted by MrXOR (2020-12-29T19:14:07 +0100; https://news.ycombinator.com/item?id=25572852 ).  
After 15 hours, 48 minutes, 53 seconds it was submitted to **Lobsters** by redecas with 10 votes and 0 comments (2020-12-30T11:03:00 +0100; https://lobste.rs/s/blu8jq/colin_s_gordon_reading_list ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Against Essential and Accidental Complexity  
URL: https://danluu.com/essential-complexity/  
First appeared on **HackerNews** with 239 votes and 91 comments, submitted by weinzierl (2020-12-29T13:37:22 +0100; https://news.ycombinator.com/item?id=25569148 ).  
After 1 hours, 18 minutes, 24 seconds it was submitted to **Lobsters** by j11g with 40 votes and 8 comments (2020-12-29T14:55:46 +0100; https://lobste.rs/s/gvahe2/against_essential_accidental ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# C Template Library  
URL: https://github.com/glouw/ctl  
First appeared on **Lobsters** with 39 votes and 11 comments, submitted by glouwbug (2020-12-30T00:25:37 +0100; https://lobste.rs/s/9gc2ku/c_template_library ).  
**Within the hour this was also posted to HackerNews!**
After 14 minutes, 54 seconds it was submitted to **HackerNews** by glouwbug with 243 votes and 115 comments (2020-12-30T00:40:31 +0100; https://news.ycombinator.com/item?id=25576466 ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  
**The same username submitted the post to both sites**.  

# Niex: Jupyter Notebooks but Using Elixir  
URL: https://github.com/jonklein/niex  
First appeared on **HackerNews** with 128 votes and 21 comments, submitted by niels_bom (2020-12-28T23:20:20 +0100; https://news.ycombinator.com/item?id=25563935 ).  
**Within the hour this was also posted to Lobsters!**
After 2 minutes, 30 seconds it was submitted to **Lobsters** by friendlysock with 10 votes and 0 comments (2020-12-28T23:22:50 +0100; https://lobste.rs/s/dnw8g7/jonklein_niex_interactive_elixir_code ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Cosmopolitan Libc: build-once run-anywhere C library  
URL: https://justine.lol/cosmopolitan/index.html  
First appeared on **HackerNews** with 589 votes and 163 comments, submitted by pantalaimon (2020-12-28T03:59:32 +0100; https://news.ycombinator.com/item?id=25556286 ).  
After 6 hours, 3 minutes, 10 seconds it was submitted to **Lobsters** by GrayGnome with 112 votes and 15 comments (2020-12-28T10:02:42 +0100; https://lobste.rs/s/xnqpyp/cosmopolitan_c_library ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Virtualize Your Network on FreeBSD with VNET  
URL: https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/  
First appeared on **HackerNews** with 74 votes and 12 comments, submitted by vermaden (2020-12-30T11:00:09 +0100; https://news.ycombinator.com/item?id=25580286 ).  
**Within the hour this was also posted to Lobsters!**
After 12 seconds it was submitted to **Lobsters** by vermaden with 2 votes and 0 comments (2020-12-30T11:00:21 +0100; https://lobste.rs/s/r5bhmo/virtualize_your_network_on_freebsd_with ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  
**The same username submitted the post to both sites**.  

# Why the iPhone Timer app displays a fake time  
URL: https://lukashermann.dev/writing/why-the-iphone-timer-displays-fake-time/  
First appeared on **HackerNews** with 220 votes and 74 comments, submitted by _antix (2020-12-28T22:56:41 +0100; https://news.ycombinator.com/item?id=25563708 ).  
After 13 hours, 28 minutes, 51 seconds it was submitted to **Lobsters** by Tenzer with 51 votes and 19 comments (2020-12-29T12:25:32 +0100; https://lobste.rs/s/yvw2xg/why_iphone_timer_app_displays_fake_time ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Contributing Without Code  
URL: https://popey.com/blog/2020/12/contributing-without-code/  
First appeared on **HackerNews** with 109 votes and 22 comments, submitted by pabs3 (2020-12-30T03:36:42 +0100; https://news.ycombinator.com/item?id=25577746 ).  
After 9 hours, 48 minutes, 21 seconds it was submitted to **Lobsters** by learnbyexample with 1 votes and 0 comments (2020-12-30T13:25:03 +0100; https://lobste.rs/s/ms9h3i/contributing_without_code ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Getting Started in BBC Basic  
URL: https://www.bbcmicrobot.com/learn/index.html  
First appeared on **HackerNews** with 68 votes and 25 comments, submitted by ingve (2020-12-29T13:36:33 +0100; https://news.ycombinator.com/item?id=25569146 ).  
After 1 days, 6 hours, 50 minutes, 23 seconds it was submitted to **Lobsters** by gerikson with 3 votes and 0 comments (2020-12-30T20:26:56 +0100; https://lobste.rs/s/fo82h4/getting_started_bbc_basic ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Implementing join planning in our open source Golang SQL query engine  
URL: https://www.dolthub.com/blog/2020-12-28-join-planning/  
First appeared on **HackerNews** with 100 votes and 15 comments, submitted by zachmu (2020-12-28T18:39:24 +0100; https://news.ycombinator.com/item?id=25561173 ).  
After 21 hours, 21 minutes, 7 seconds it was submitted to **Lobsters** by eatonphil with 3 votes and 0 comments (2020-12-29T16:00:31 +0100; https://lobste.rs/s/a6suos/planning_joins_make_use_indexes ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Teaching the Unfortunate Parts  
URL: https://www.executeprogram.com/blog/teaching-the-unfortunate-parts  
First appeared on **HackerNews** with 123 votes and 90 comments, submitted by gary_bernhardt (2020-12-29T00:37:26 +0100; https://news.ycombinator.com/item?id=25564666 ).  
After 7 hours, 22 minutes, 49 seconds it was submitted to **Lobsters** by Hail_Spacecake with 9 votes and 6 comments (2020-12-29T08:00:15 +0100; https://lobste.rs/s/4ptsq9/teaching_unfortunate_parts ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  

# Running BSDs on AMD Ryzen 5000 Series - FreeBSD/Linux Benchmarks  
URL: https://www.phoronix.com/scan.php?page=article&item=amd-5900x-bsd  
First appeared on **Lobsters** with 3 votes and 0 comments, submitted by vermaden (2020-12-30T11:02:18 +0100; https://lobste.rs/s/fmooqn/running_bsds_on_amd_ryzen_5000_series ).  
**Within the hour this was also posted to HackerNews!**
After 5 seconds it was submitted to **HackerNews** by vermaden with 77 votes and 42 comments (2020-12-30T11:02:23 +0100; https://news.ycombinator.com/item?id=25580298 ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.  
**The same username submitted the post to both sites**.  

# Bash HTTP Monitoring Dashboard
URL: https://raymii.org/s/software/Bash_HTTP_Monitoring_Dashboard.html
First appeared on **Lobsters** with 30 votes and 2 comments, submitted by raymii (2020-12-27T13:58:40 +0100; https://lobste.rs/s/4pivy1/bash_http_monitoring_dashboard ).
**Within the hour this was also posted to HackerNews!**  
After 5 minutes, 36 seconds it was submitted to **HackerNews** by todsacerdoti with 160 votes and 26 comments (2020-12-27T14:04:16 +0100; https://news.ycombinator.com/item?id=25550732 ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.

3 posts appeared first on Lobsters and 11 posts appeared first on HackerNews.
Average time for a cross-post: 8 hours, 39 minutes, 55 seconds .
Average comments on HN: 64, Lobsters: 4.
Average score on HN: 212, Lobsters: 22.

The test option is mostly for me to figure out time stuff in different time zomes (looking at you, github actions):

--- START TEST ---
Date/time/timezones are hard. Below is a test post comparison,check if your timezone information is correct. The difference between Lobsters and HN should be 5 minutes and 36 seconds.
Number of posts from Lobsters    : 1
Number of posts from Hacker News : 1

Matches (1):

# Bash HTTP Monitoring Dashboard
URL: https://raymii.org/s/software/Bash_HTTP_Monitoring_Dashboard.html
First appeared on **Lobsters** with 30 votes and 2 comments, submitted by raymii (2020-12-27T13:58:40 +0100; https://lobste.rs/s/4pivy1/bash_http_monitoring_dashboard ).  
Then, after 5 minutes, 36 seconds it was submitted to **HackerNews** by todsacerdoti with 154 votes and 26 comments (2020-12-27T14:04:16 +0100; https://news.ycombinator.com/item?id=25550732 ).  
The highest score was reached on HackerNews and the most comments were on HackerNews.

1 posts appeared first on Lobsters and 0 posts appeared first on HackerNews.
Average time for a cross-post: 5 minutes, 36 seconds .
Average comments on HN: 26, Lobsters: 2.
Average score on HN: 154, Lobsters: 30.
--- END TEST ---

The new option isn't very useful (very volatile though), but still fun to look at:

Which stories appear both on Lobsters and on HN, who was first?
An excuse to play with parsing a JSON api in C++ with async by Remy van Elst (https://raymii.org)

Current date/time: 2020-12-30T22:28:23 +0100

Fetching HackerNews New Stories async (200 posts) (https://github.com/HackerNews/API)
Fetching the first ten Lobsters pages (/newest) async 10*25=200 posts) (https://lobste.rs/s/r9oskz/is_there_api_documentation_for_lobsters_somewhere)

Number of posts from Lobsters    : 200
Number of posts from Hacker News : 189

Matches (3):

# Two Concepts of Legibility
URL: https://ideolalia.com/essays/two-concepts-of-legibility.html  
First appeared on **Lobsters** with -1 votes and 0 comments, submitted by ztellman (2020-12-30T20:40:48 +0100; https://lobste.rs/s/0mel38/two_concepts_legibility ).  
**Within the hour this was also posted to HackerNews!**
After 4 minutes, 28 seconds it was submitted to **HackerNews** by todsacerdoti with 2 votes and 0 comments (2020-12-30T20:45:16 +0100; https://news.ycombinator.com/item?id=25585205 ).  
The highest score was reached on HackerNews and the most comments were on nowhere.

# Unsafe String Interning in Go
URL: https://mdlayher.com/blog/unsafe-string-interning-in-go/  
First appeared on **HackerNews** with 3 votes and 0 comments, submitted by mdlayher (2020-12-30T18:58:16 +0100; https://news.ycombinator.com/item?id=25584046 ).  
**Within the hour this was also posted to Lobsters!**
After 6 seconds it was submitted to **Lobsters** by mdlayher with 2 votes and 0 comments (2020-12-30T18:58:22 +0100; https://lobste.rs/s/jefzu1/unsafe_string_interning_go ).  
The highest score was reached on HackerNews and the most comments were on nowhere.  
**The same username submitted the post to both sites**.

# A little puzzle with printf() and C argument passing
URL: https://utcc.utoronto.ca/~cks/space/blog/programming/PrintfAndArgumentPassing  
First appeared on **Lobsters** with 13 votes and 7 comments, submitted by calvin (2020-12-29T05:35:40 +0100; https://lobste.rs/s/jkdzu9/little_puzzle_with_printf_c_argument ).  
After 1 days, 13 hours, 33 minutes, 32 seconds it was submitted to **HackerNews** by emptybits with 1 votes and 0 comments (2020-12-30T19:09:12 +0100; https://news.ycombinator.com/item?id=25584196 ).  
The highest score was reached on Lobsters and the most comments were on Lobsters.

2 posts appeared first on Lobsters and 1 posts appeared first on HackerNews.
3 posts appeared first on Lobsters and 2 posts appeared first on HackerNews.
Average time for a cross-post: 8 hours, 17 minutes, 21 seconds .
Average comments on HN: 1, Lobsters: 1.
Average score on HN: 5, Lobsters: 3.
Tags: c++ , cmake , cpp , development , hackernews , json , lobsters , network , software