Asynchronous Programming with JavaScript Part 2: The Amazing Callback

Category asynchronous programming, callbacks, JavaScript

This is part two of the Aysnchronous Programming with JavaScript series.

In this part, we discuss the Callback solution with explanation and some progressively complex examples.


Doing a Hackathon to Get Your Project Across the Finish Line

Category Development, Hackathon, workplace

Sometimes deadlines can be tight. Either time slips away and the progress gets behind on a project, or possibly enough time isn’t given up fron for your team to get anything donet. Or perhaps you have a great app idea that you want to make happen and get out right away. No matter the situation, maybe the right solution is a hackathon.


Parsing Large JSON with NodeJS

Category
Recently I was tasked with parsing a very large JSON file with Node.js   Typically when wanting to parse JSON in Node its fairly simple. In the past I would do something like the following. Or even simpler with a require statement like this Both of these work great with small or even moderate size files, but what if you need to parse a really large JSON file, one with millions of lines, reading the entire file into memory is no longer a great option.   Because of this I needed a way to “Stream” the JSON and process as it went.  There is a nice module named 'stream-json' that does exactly what I wanted.   With stream-json, we can use the NodeJS file stream to process our large data file in chucks.  Now our data can process without running out of memory, however in the use case I was working on, inside the stream I had an asynchronous process.  Because of this, I still was consuming huge amounts of memory as this just up a very large amount of unresolved promises to keep in memory until they completed.   To solve this I had to also use a custom Writeable stream like this. The Writeable stream also allows each asynchronous process to complete and the promises to resolve before continuing on to the next, thus avoiding the memory backup.    This stack overflow is where I got the examples for this post.   https://stackoverflow.com/questions/42896447/parse-large-json-file-in-nodejs-and-handle-each-object-independently/42897498   Also note another thing I learned in this process is if you want to start Node with more than the default amount of RAM you can use the following command. By default the memory limit in Node.js is 512 mb, to solve this issue you need to increase the memory limit using command --max-old-space-size.  This can be used to avoid the memory limits within node. The command above would give Node 4GB of RAM to use.  

Keys to Workplace Wellness

Category wellness, workplace

In today’s world wellness is typically equated to physical exercise and what kind of workout regimen a person keeps.  However, from my perspective, this is an overly simplistic definition.  To have a well rounded approach to wellness we must consider mental, emotional, spiritual, and physical well-being.  Yes, all of these directly impact our day-to-day work place and overall life experience.  In part one of this three part blog we will cover Mental/Emotional well being.


Relocating Ticket Work via GIT

Category GIT, SourceTree, Tips

Mis-referencing work on branches and feature branches happens.

GIT provides renaming and history-altering tools to help us fix our silly mistakes.

This post runs through a rather simple way to relocate work from one ticketing reference setup to another using SourceTree GIT management tools.


Using Postman Environment Variables to Help Make Ease of API Development

Category APIs, Postman

When developing or testing APIs, Postman makes the job simple for us. There are many tools available that we can use to do our development or testing with ease. What I found to be most beneficial are environment variables. If you are working with many different environments, you may set up each with any authorization tokens, urls, or any other variables you find necessary to set. After setting up your variables and building your request, all that’s left is to select your environment and make your call.


Couchbase - Big OFFSETs can be a big problem

Category Couchbase, N1QL, NoSQL, SQL

Dealing with large data sets and big data can often call for new techniques to optimize performance.  When you have a few million documents or rows, you can do things differently than when you have to deal with tens of millions of rows or documents in either a standard SQL database or a NoSQL database like Couchbase.  One thing we noticed at CKH is that in very large document stores, the OFFSET and LIMIT style pagination on both N1QL queries and Views in Couchbase was getting slow as the OFFSET grew larger.  This becomes a problem when you want to build efficient, consistent paging into your system.  Not to worry, Couchbase provides a solution.


Mutli-stage Docker Builds

Category DevOps, Docker

Docker is a great tool for deploying scalable web applications for a large variety of technologies ranging from Node.js, Python, MySQL, and more.  We have been using Docker at Computer Know How for some time but we recently ran into a challenge with an application we were setting up for Docker.


Using insertDesignDocument to Add Views

Category Couchbase, insertDesignDocument

If you have multiple views to set up, this is something that should not be overlooked. It is well worth the time setting up a service to import all of your views automatically. It will save both yourself and others the pain of manually adding views in Couchbase. 


Scripting: Work Smarter, Not Harder

Category Python

Have you ever been given a task that seemed so repetitive or manual that you just knew there must be a better way to accomplish the task? If your answer is yes, then I have been in your shoes before. Anytime I am given a project that involves manually searching through files, converting data from one format to another, or pulling reports from data that isn’t setup for reporting, I ask myself some questions such as "how could we accomplish this quicker?" or "could this task be scripted?"


1 2 3 4 >>