Recently, during a project I was working on, we were given a requirement:
— “To retrieve records from the DB within 10 milliseconds consistently, from two large data sources/tables.”
One data source contained approximately 686 million records and the other contained 24 million records.
On top of this, the requirement stated that the preference for this solution should be towards an AWS managed DBMS, so the database instances can be easily managed in the long run.
Thus, with these requirements, the hunt for the best-suited DBMS began. 🤞🏽
With the data volume being huge, and the read rate high (about…
I have collected the crime data of past 8 years from the official Srilanka Police website: Police crime statistics
As all the data are in PDF format it is converted into CSV using python Tabula library. Then missing data and corrupted cell values during the PDF to CSV conversion were fixed with excel.
from tabula import convert_intoconvert_into("distribution_of_grave_crime_abstract_for_the_year_2014.pdf", "distribution_of_grave_crime_abstract_for_the_year_2014.csv", pages="all")
Even though the Population of Sri Lanka is increased by 7.278 %, the overall crime in Srilanka over the past 8 years is reduced by 36.8398%. That’s great news for all Sri Lankan Citizens. …
Python downloads its dependencies from PyPI repositories by default. It contains latest versions (can be stable or not) and various amount of packages. We’re good right? So, whats the need of custom private package repository?
You can control dependencies of packages explicitly regardless of versions deprecation or latest backward incompatible versions. Of course, this can be done by defining versions in requirements.txt, but to make sure that every package we get is as we expected its better to use custom repository.
PyPI servers have discontinued support for TLS 1.0 / TLS 1.1 version devices to download packages.
“I am going…
The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka. This can be used for example as a queue between several syslog-ng instances.
Kafka source is my Google summer of code project for the year 2016 with the Syslog-ng organization under the guidance of Viktor Juhász. Syslog-ng can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to the destinations.Syslog-ng has a Kafka destination, which is implemented in Java. Reading messages from kafka will make it as a queue between two Syslog-ng instances.
Kafka is a distributed messaging system provides fast and highly scalable and redundant messaging through a pubsub model. It allows a large amount of adhoc consumers and also it self heals from…
It’s my third and final project during my internship at WSO2. It is a property of software that enables a system to perceive that it is not operating correctly and, with/without human intervention, make the necessary adjustments to restore itself to normalcy. I read a few research papers since this area is still new to the industry, I couldn’t find any actual implementations. Therefore, I had to spend a lot time reading research papers to get an idea about self-healing software.
Fault tolerance computer systems mirror all their operations, so if one component fails another redundant component will take part…
Computer Science Engineer