Start by creating a new Google Doc, and name it “Template – 1:1 Notes”.

In the Google Doc, write a new heading, “Week of” and insert a date smart chip and set it to today’s date. Whenever creating a new heading, you can update the date chip to when the 1:1 meeting took place. In the text for the section, write out two lists for the “Topics” and the “Action Items”. It should look like this:

Week of May 1, 2026

Topics:
-

Action Items:
-

For the topics, you can add recurring questions or topics.

Then you will select all of that text and create a custom building block by right-clicking on the content and clicking “Save as custom building block”. Give it a name such as “1:1 Notes”.

For each team member, save the template as a new copy, rename it to include the team member’s name, and share it with them. When it is time for the next meeting, insert the custom building block, updating the date to the date of the meeting. The action items can be assigned to you or the team member.

Organizing and archiving 1:1 meeting notes is simple. Use a tab in the Google Doc for each quarter or each year, and start a new tab when it is the start of the next quarter or a new year.

Getting Buy-in for Lattice

The Lattice tool has four packages, add-ons, and enterprise pricing. All of the subscription packages include:

• 1:1s notes
• Weekly Updates to ask the team questions every week and to gauge the sentiment of the team
• Feedback to share feedback with the team and with peers
• Q&A Boards for employee questions

To gain buy-in for Lattice, one idea is to buy enough Lattice seats for one team. Another idea is to buy it for a few teams or a department and select the package that includes 1:1s and has the immediately useful features, such as the “Engagement” package with pulse surveys and employee Net Promoter Score (eNPS). Depending on meeting cadence, this can be done for one month or one quarter in order to evaluate Lattice.

One viable competitor has emerged: Rspack. It is a Rust-based build system for JavaScript and TypeScript web apps that is compatible with Webpack. Rspack works with many Webpack plugins and offers replacements for the major Webpack plugins. There is a documented migration path from Webpack to Rspack. By targeting compatibility with the major Webpack APIs, and with good documentation, Rspack reduced the complexity and the effort and made migration feasible.

Webpack Performance Issues

The search for a faster, better build system was started because of performance issues with webpack. The issues were the speed of compilation and the memory usage required. The team put time into researching and investigating the performance bottlenecks in webpack, and I have written about webpack build-time performance for React.

With newer MacBooks, the speed of webpack was faster due to the hardware. However, the React app was also being built and re-compiled within a Docker container. In the CI process, we could pay more to increase the memory available, and wait for webpack to finish the build. For a local MacBook Docker or cloud-based Linux development environment, the slowness affected the workflow of engineers. Due to a few plugins and a few cycle issues with imports, memory usage for a webpack build required a lot of RAM, up to 12 GB. For engineers, they would see a spike in memory usage, and subsequent iteratively compiled bundles had smaller spikes of memory usage. Many engineers opted for building the web app once and not having hot-reload available within Docker, or building it on the host machine which was slightly faster.

The variance in environments, from the latest MacBook Pros to Linux virtual machines with limited CPU and memory, required testing in each environment to measure the baseline performance. When we migrated to Rspack, testing each environment was a necessity to avoid regressions and to benchmark the performance of the build. In the CI/CD environment, we could check the CPU and memory usage across multiple jobs and workflows, along with the cost of the build/test run.

What are the benefits of Rspack?

Let’s begin with the technical outcomes. The technical outcomes of switching to Rspack:

• Faster initial and iterative compilation
• Lower memory usage

Now, the benefits of rspack:

• Engineers can work on React web app features smoothly and iterate and deliver sooner
• Designers, QA, product managers can try out and test features sooner
• Lowers the cost of CI/CD service by reducing the memory required and shortening the build time

You may be wondering why I split the benefits into two. Well, it’s because the technical outcomes are usually what engineers state first, and they assume the conclusion that the technical outcomes are desirable and the effort is worthwhile. As an engineering manager, I like seeing the benefits that will lead to a business outcome, and the drawbacks that may affect other teams. The technical outcomes are what realize those benefits. The benefits show what the technical outcomes lead to. With a slower build system, engineers deliver slowly. With high memory usage, the costs of the CI/CD service are very high. By saving time and money we ship sooner and can re-allocate costs from CI/CD to another line item in the budget. Sometimes the benefit is enabling whole new features or even product lines because specific technical challenges were solved.

Planning and Implementing Rspack to replace Webpack

The planning process outlined the following:

• Where we built and deployed the React web app (CI/CD, local and cloud-based development environments)
• The set of webpack plugins we use as part of the build
• What constituted a go-ahead in the prototyping phase
• The test scenarios for a production rollout
• How we could rollback to webpack in case of any issues
• The stakeholders who would need to sign off on the change, from staff frontend engineers to QA to directors of engineering.

In the prototyping phase, we wanted to see how far we could get by replacing webpack with rspack, then by replacing webpack plugins with rspack equivalents and building the bundle locally. With the findings documented, we moved to the next part of the prototype, running rspack and webpack side by side in the CI build pipeline. After that, it was a wider test by the QA team to check for regressions, and more engineers trying it out locally and in Docker.

When we were certain that Rspack could build the bundle, that there were no test regressions, and that we had benchmark data showing an improvement, we moved to implementation.

What are Cache Layers?

Caches are a way to improve performance and reduce latency by trading space for time. The simplest example to start with is when you have to recalculate a value in a recursive function such as the Fibonacci sequence. Instead of calculating the previous results over again, they are stored in a hash table or dictionary. Caching is sometimes called memoization. For a website where the content does not change frequently, the server can cache the content in memory and serve it again while the client, the web browser, can also cache the content before making a request back to the server.

Layered Mental Model

In the cache layers model, we are adding options when caching. There are three layers: in-memory/in-process cache, sidecar/local node cache, distributed cache.

The in-memory cache is the first layer and for many non-distributed programs it is the only layer. More intensive programs will have two layers: in-memory and sidecar/local cache. These programs are not distributed and will run a persistent cache across multiple runs of the program, it can be sqlite, redis, or a file with a custom format. Distributed applications add a third layer, the distributed cache.

Latency Budgets

When a user is accessing a part of the system, they have an expectation of a fast response. When they click a button, they expect something to happen within milliseconds or seconds, depending on the application and the action. Latency budgets can be set for core actions in the system to ensure that they are meeting expectations.

Let’s understand through an example. A user is filtering and sorting and searching through a table of data. There are 1 million items in the table, only 100 items are displayed on the page. The latency budget for sorting is < 0.5 seconds, since it is the same data, unchanged and only the sort order changes based on a selected column. The latency budget for filtering is slightly higher, < 1 second is the expected time and can scale with the amount of items, based on the user’s perception, and the latency budget for searching could be < 5 seconds. Let’s consider another dimension, newness of the item. If users are frequently searching for new items, the latency budget for those new items could be as low as < 0.5 seconds. For older items, users expect < 10 seconds, a lot more room. The expectation and perception depend on multiple factors including a user’s past usage of similar systems, their usage of the current system, and the user interface of the system.

When setting a latency budget, you can start with an observed baseline and add a percentage for the budget, and subtract a percentage to set as a goal.

Caching With Layers for a Web Service

In a real-world production web service, caching can be used in many ways for a variety of reasons. More complex web applications not only have a backend API server, they also have background workers. For example, there can be very expensive database queries that are used frequently, or there can be logic and calculations that do not change often.

L1 In-Process Cache

An incoming web API request can require multiple expensive database queries in order to return a response. The memory within a web server process can be used as a cache. For example, in Python, there is a @cached decorator function that will create an in-memory mapping between the parameters of the function it wraps and the resulting value. This can be used at different points along the code path. If another API endpoint also uses those functions, then the cache will be used. We can expand on layer one caching by caching more parts of the request, or by caching the whole request/response. When we request /items/123 repeatedly, the request and response can be cached in-memory.

The limitation of only layer 1 in-process/in-memory caching is that this caching only works for one process. If the web server runs 4 processes, the typical setup does not use sticky routing, which means a user request may go to any of the 4 processes and suffer from a cache miss. As more requests come in, the likelihood that all 4 process caches have the value is increased. One way to workaround this limitation is to pre-warm the cache, though this is very dependent on user request patterns and may require a lot of memory.

The other limitation is that the in-process memory cache is it increases the memory required for each process. If the web server initially requires 1GB, it could grow to double or more in order to have enough in-memory cache capacity. A cache key TTL and expiration time is required when using this type of caching. The in-memory cache competes with the rest of the program for memory.

L2 Sidecar / Local Node Cache

The second layer of caching requires running Redis (or Valkey another cache) as a sidecar or as shared service.

Continuing with the example, the web server runs 4 processes and we run only 1 web server. However we are scaling up to 3 web servers for a total of 12 processes. With a side-car, each of the web servers will have its own cache shared by 4 processes. Another option is to run the cache as a daemon service on each compute node. When vertically scaling, the additional web servers will make use of the cache already existing on the compute node. When horizontal scaling, new nodes will have a fresh cache so initial requests on the new node will be slower.

L3 Redis, Memcached

Using a separate cache server such as Redis or Memcached is the third layer of caching. The advantage is that the cache can persist between deployments of services and compute nodes, it is a de-coupling of the cache from the rest of the services.

In the example, we have multiple nodes running multiple web servers with many processes. By having this third layer, the first request to a web server can cache the request and response, and it can be re-used no matter if we scale to 1000 web server processes or down to 1 server.

The third layer of caching can be applied to storing serialized objects from database queries, storing HTTP responses, and more. Pre-warming the cache on deployments becomes more valuable especially if there are common access patterns from particular customers based on past analytics data.

Since DevOps is an IC role that works with many teams across an organization and can have varying types of responsibilities depending on the organization, the questions are related to those areas. My assumption with these questions is that you, as a DevOps manager or hiring manager, already will be asking about core DevOps skills such as whether the candidate knows what CI/CD means, the main services in the major cloud service providers, and how to use IaC and Configuration Management tools. In each question, you can also ask about observability and security, who should have access to what and what kinds of preventative alerts are needed.

We’re setting up a prototype sandbox environment to test a product feature with a prospective client, how should we do this?

For a B2B SaaS or company with enterprise customers, the sales team will often need a sandbox environment to test a new feature for a potential new client. Typically this is a manual process at first, and then later automated.

An experienced DevOps candidate will ask about automation, and will suggest using Terraform and Chef or Ansible or another Configuration Management tool in order to set up the environment. They should be able to outline the steps to do this, such as setting up a new machine image or an Ansible playbook to run to set up the machine.

Candidates should also ask about supporting infrastructure, such as cache and database infrastructure that are needed for the test environment.

How can we ensure our web app or site is working?

This question demonstrates whether a DevOps engineer candidate knows what an automated canary is and how the QA process for a deployment works and ask questions about that process. DevOps engineers need to at least know how to manually test, using tools such as curl or Yaak to hit API endpoints directly, or to use a browser inspector to check for JavaScript errors. Experienced DevOps candidates will also mention uptime monitoring services.

We’re setting up a new micro-service, how should we set up CI/CD?

Candidates should demonstrate that they understand continuous integration and continuous deployment by suggesting GitHub Actions, CircleCI or some other service provider. They should ask whether container images need to be created or if it’s an AWS Lambda (or equivalent) function-based codebase that needs to be deployed.

The candidate should demonstrate knowledge about their expertise in the particular cloud stack for a deployment, such as the container registry and the observability tools. The candidate should also mention Grafana, Prometheus, and other industry standard tools.

We are using a specific AWS/GCP/Azure technology, and we want to build a resilient architecture, how would you go about doing that?

Many organizations have a primary cloud service provider, such as AWS. For high availability and redundancy and resilience, it can be advantageous to load balance and service requests across cloud providers or to set up backups in case of downtime from the primary provider.

Asking this question allows the candidate to show whether they think about multiple regions within one service provider or across service providers.

A candidate will typically be well-versed in major cloud service provider, and will show that they understand what options there are within the provider for high availability. So even if not considering moving from AWS to Azure, or balancing a CDN across CloudFlare and AWS, it gives the candidate a chance to show they know what it takes to run a high-availability service.

A team needs a new database set up, how would you plan out this new infrastructure?

Every cloud-based software product starts with a database, whether it’s Postgres, MySQL, MongoDB, Firebase, or something else. Sometimes a new product or new micro-service is being built that will need its own database.

The DevOps engineer candidate should know what questions to ask before they dive into infrastructure planning, and should ask questions about read replicas, database instance types and database size, and about backups and snapshots. They should also ask about tear-down and upgrade situations, for example if the new micro-service moves into production do we need to upgrade the size and instance type of the database? How should we tear it down to re-allocate spend and technical resources?

What is your process for working with other teams?

Start with this initial question, “what is your process for working with other teams?” and have the DevOps engineer candidate apply it within a scenario:

• A team needs urgent infrastructure work completed, apply your process to it.
• The web app is suffering an outage, and a product manager and staff engineer have started a meeting. What steps would you take to resolve the issue, according to your process?
• A team is planning out a project that is expected to start next quarter. You have already met with the tech lead who architected it. However, another team with similar infrastructure requirements is starting a project sooner than that.

The “process” is important because there will always be something coming up that may need the urgent attention of the DevOps team and there will be many tickets coming from internal teams. How will the candidate respond in those situations? How will they pause work they’re currently doing? Who will they communicate with to inform them of issues or of status updates?

You are on-call, and there is an urgent issue, what steps do you take first?

In many organizations, the DevOps team’s responsibilities includes SRE, Site Reliability Engineering. The team has an on-call rotation and is part of the first line of defense when there are issues that show up in production. The urgent issue can be specific to your product, or it can be general, such as email notifications or authentication systems not working, or a particular external API is failing.

The goal of this question is to understand how the candidate will handle the pressure of on-call. One of the first steps could be looping in managers with a status update, or posting a public status update for customers rather than diving right away into debugging. Another first step could be diagnosing and triaging and prioritizing the issue, and asking if there’s a way to patch it or roll a code or infrastructure change back and buying time for engineers to find a more permanent fix. In smaller organizations and startups, the first step could be debugging and posting updates as they go along.

After an urgent issue is resolved, what steps do you take next?

A follow-up question to the one above is, what happens the next day? The goal is to understand whether the DevOps engineer will write up notes, research ways to prevent the issue, create alerts to prevent the issue, and talk to other engineers to create a better process for the future.

The question is a bit free-form because it depends on the DevOps engineer. Each one knows what steps they would take based on their experience, and having it free-form lets you see how well it aligns with current processes. One DevOps engineer may focus mainly on the urgent issue and the technical details while another may focus more on preventative measures.

Within this question you can also ask about observability and systems monitoring.

We have a new build pipeline being set up, how do we handle secrets management?

This question digs into the security side of DevOps. A candidate should know that API tokens and secret keys should be stored outside of a Git repository, and should not be stored in plain-text, and should be stored in AWS Secrets Manager, 1Password, or other secrets management tools. They should be able to describe how those secrets are accessed in the build pipeline and how they’re accessed for the deployed application. Candidates should also ask about key rotation and different keys for different environments.

Time-boxing these spikes, or explorations, with finite time limits allows for spending some time exploring ideas, different approaches, and different implementations. A time box suggests that certain information should be gathered within three days, two weeks, or some other time limit before that approach is abandoned or accepted. However, this is only one of the tools that you can use to account for unknown time/effort.

As a project progresses, there can still be many unknowns and some uncertainty. The “Cone of Uncertainty” is a helpful concept to visualize the level of uncertainty in a project,

The Cone narrows only as you make decisions that eliminate variability. […] defining the product vision (including committing to what you will not do), reduces variability. Defining requirements—again, including what you are not going to do—eliminates variability further. Designing the user interface helps to reduce the risk of variability arising from misunderstood requirements. Of course, if the product isn’t really defined, or if the Product Definition gets redefined later, then the Cone will get wider, and estimation accuracy will be poorer.

Two proactive tools to manage this uncertainty are to use buffers and placeholders which address specific uncertainties by making time to handle them. The concept of the “buffer” is relatively underused in Agile and Agile SCRUM methodology. Many projects plan out specific tickets for work to be done, but there are still some unknowns.

Placeholders are tasks that say, “We need to investigate this,” “We need to do some research here,” or “We need to monitor and check in on some data.” They can be useful for marking maintenance or check-ins, re-evaluating past decisions, and adjusting the path forward. The benefit of the placeholder is that it ensures that critical parts of the project are completed. The placeholder is a simple check if the task is no longer relevant or has already been accomplished. If there’s still work to be done, the placeholder ensures there was time and effort available in planning to achieve that work.

A buffer is an additional task attached to a primary one. It is similar to time-boxing, where the buffer says we need to explore a specific idea. In most projects, this can take a few days to a few weeks. When estimating a long-term plan, this buffer can be used to extend or shorten the timeline. For example, software development projects sometimes involve setting up new infrastructure; however, it may be possible that at least some of the infrastructure already exists.

Buffers are a source of time and effort for conducting research, investigation, re-evaluations, check-ins, and monitoring. Each time a task related to the buffer is accomplished, the buffer becomes smaller, with less time and effort available for the subsequent investigation, prototype, or check-in. The benefit of the buffer is that it sets expectations on the time, effort, and scope of a project.

Calendar: Google

The most essential tool is the calendar, and Google Calendar is great. As a manager, creating recurring events such as daily standups or bi-weekly sprint planning meetings is standard.

Creating and subscribing to multiple calendars is a good feature. Depending on the number of projects and teams, creating a separate calendar for recurring events can make sense. Many organizations have a separate calendar for engineering department meetings and presentations. Any meetings affecting multiple departments or teams within a department could be on a shared calendar. Office hours are another candidate for a shared calendar across the organization. Multiple shared calendars are useful in keeping the main calendar manageable.

For ICs, setting up “focus time” in the calendar is a good way to let engineering managers know when they can schedule meetings to avoid interrupting focused work time. Subscribing to multiple calendars is also great for keeping track of holidays and out-of-office time, which can affect capacity planning and on-call schedules.

Tasks, to-dos: Todoist, Google Tasks

There are many task management and to-do list tools out there. Previously, I used Todoist for task management and tried out a few different tools.

The best to-do list is close to the other tools you use for managing a team.

If you live in Google Calendar, then Google Tasks is a good choice since it integrates directly into the calendar, and like calendars, you can create multiple to-do lists. You can set tasks to repeat, a good reminder that appears in the calendar without taking up a block of time. Tasks are a good reminder for various preparation work for project planning and roadmapping.

Notes: Notion, Obsidian, Google Docs, Slack

For short-form notes, what I’ve found works best is using whichever tool is available. If I’m in a meeting on Google Meet or looking at the calendar, I can use Google Keep for notes. If I’m in a Slack huddle or see something in a channel or thread that I want to write a note about, I use the “Direct Message to self” feature of Slack as a note-to-self. I use Notion or Obsidian for long-form notes or for turning short notes into long-form ones. Writing notes and creating links between them is very useful. The links between notes make it easy to refer to past notes about meetings or additional notes about documents.

One of my workflows involves using Slack and Notion. I use Slack for a summary that can be copied/pasted to various channels and threads and Notion for a longer-form note with more details that I rewrite later for different formats (e.g., a technical spec, engineering brief, a company-wide presentation, or during a 1:1). The same applies to Teams, Google Docs, or other tools.

Obsidian Plugins

There are many plugins for Obsidian to help you with note-taking and creating a knowledge base:

• Checklist: This plugin gathers all checklists, to-do lists, and tasks across all pages and puts them in the sidebar. It makes it easy to create checklists in daily notes and keep track of which ones are completed later.
• Note Refactor: This plugin enables you to select text and extract it into a new page. Do you have a long page with many notes about different topics written while you were in the flow? That’s no problem; you can branch out into new pages with the Note Refactor plugin.
• Periodic Notes: This plugin will create weekly, monthly, quarterly and yearly notes for you at the click of a button. This makes it easy to summarize your week or month. As an engineering manager, I found that a monthly list of what the team got done last month and what was upcoming was great for communicating the value the team was delivering. The quarterly notes can tie into organizational goals.
• Iconize: add icons to files and folders; this plugin is good for highlighting very important pages with a visual cue.

1:1s and Career Goals: Lattice

Lattice is incredible for this. The software formalizes what’s usually in spreadsheets or in document templates. For 1:1s, I encourage ICs to add topics; it’s a checklist to ensure we cover each topic. There’s also a checklist of “action items” for the IC and myself to ensure we follow up with concrete actions based on our conversations. Another great feature is the career track and career goals. The org can set up the templates for the career track, and I can work with an IC to turn those into specific goals. Lattice makes it far easier to collaborate with an IC on their goals and to ensure there’s accountability from myself and the IC on accomplishing things that not only move the company or projects forward, they move the IC forward in their career.

Diagrams: Lucidchart, Drawio

Lucidchart and Drawio are great diagramming tools. Lucidchart has many great templates to use as starting points, such as user journeys, use case diagrams, sprint retrospectives, roadmaps, and cloud architecture diagrams. Drawio also contains many great templates and can be downloaded locally. I primarily use Drawio for engineering architecture diagrams.

Both Lucidchart and Drawio can be used for quickly sketching out diagrams. The user interfaces are well-designed. Lucidchart has a good collaboration mode, where you can share a diagram and build it out with your team.

SQL Database and Codebase Tools: JetBrains

JetBrains suite for professionals — from IntelliJ IDEA to PyCharm to DataGrip, each tool makes it easy for engineering managers to explore and understand a code base, data models, and database schema.

I use IntelliJ IDEA to get up to speed on a codebase by searching the code or clicking through to see the usage and definitions of methods and classes. The integrated debugger works great for Python and Ruby codebases, and the autocomplete and code navigation for TypeScript codebases are fantastic.

The database tools in JetBrains IDEs allow you to connect to various data sources, from DynamoDB to PostgreSQL to ClickHouse to Redis and SQLite. The tool lets you run queries and edit records and rows within the editor, which is good for checking data and modifying it during testing. It’s also possible to sort, filter, and search the data locally and to compare data.

While there are other database tools, having an integrated database viewer in your IDE that sits beside the code that creates and updates that data is very useful.

Password Management: 1Password, Bitwarden

1Password and Bitwarden are solid choices for password management. Both feature vaults that can be shared with team members and across departments. They both feature SSO, which enables your IT department to control access to the vaults. Bitwarden is also open source, with the core code licensed under the GPL.

For software engineers, IT, and DevOps, the CLI, and APIs offered by 1Password and Bitwarden allow for the automation of secrets management and vault management.

React and Performance

React is an excellent framework for building web apps combined with other libraries.

However, there are multiple parts to managing and improving the performance of a React app:

• State management libraries, such as Redux or MobX or React Query, can affect runtime performance. They control when re-renders occur and when and how often data is retrieved.
• Type-checking with Flow or TypeScript can affect the build-time performance.
• webpack has loaders and module resolution that can affect build-time performance (other build systems may have different performance considerations).
• Caching in webpack can reduce build-times on subsequent builds for development or be disabled for production builds.
• Webpack can generate source maps at different levels with different effects on build-time and runtime performance.
• Targeting multiple runtimes, such as web and mobile (React Native) and desktop (Electron), can impact the build-time performance and the runtime. For instance, it may be acceptable on the desktop to load a large bundle JS with all the app components, whereas on the web, it’s better to lazy-load and load the app’s components as needed.

It’s also essential to understand the usage pattern of the app and what kind of effect longer initial page load times will have. Are your clients and customers accessing the app at the beginning of their workday and using it throughout the day? In that case, a large bundle size is okay. Are users accessing it throughout the day but only using a few features or pages of the app? A small initial bundle that includes those commonly accessed pages with everything else lazy-loaded is the way to go.

Runtime performance has to be weighed by understanding the impact on users, the amount of effort to improve and maintain it, and the business value derived from it. Build-time performance is a set of trade-offs that includes the developer experience.

When debugging memory issues, you have a few levers that you can control in the build process, such as the configurations and the plugins and loaders, and there are layers to the memory issues. Some of the problems may be in your webpack configuration; other times, the issues are deeper at the level of a function in core webpack or an extra plugin.

Build-Time Performance for webpack

Build-time is all the steps required to build a bundle of your web app. From TypeScript compilation to SASS and PostCSS compilation to source map generation to minification. You can measure build-time performance along two dimensions: timing and memory usage.

CPU Profiling Plugin and Timings

webpack’s ProfilingPlugin generates a JSON file showing which functions were called, how often, and how much CPU they used. In addition, they are grouped by the webpack plugins so you can see which ones are taking up the most time during the build process. Improving the overly slow parts is only possible with profiling data that identifies where the build-time performance bottlenecks are.

You can analyze the JSON file in Chrome Dev Tools’ Performance tab. However, larger builds may create JSON files over 100 MB, too large for Chrome Dev Tools. If the file is large, you can use 0x to view a flame graph. However, 0x may not display all the information. Instead, you can use the Node.js CPU profiling with webpack.

Heap Memory Profiling

heap-sampling-webpack-plugin is a plugin for webpack that uses the Node inspector to snapshot the build process’s memory usage. It is simple to set up, and it generates heap profile files. A heap profile contains a tree of function calls and the memory used and retained. You can view the heap profile files in the Memory tab of Chrome Dev Tools or VS Code. Both tools provide a tree of function calls to dive deep and discover where memory is being allocated and retained. Both Chrome Dev Tools and VSCode can display a flame graph, which is a more compact view and can be easier to navigate.

Here’s a video demonstrating how the heap profile looks like in Chrome Dev Tools:

It’s a very neat and practical plugin that reports peak memory usage, vital for continuous integration environments.

When profiling the memory usage, you must consider whether that part of the build process is necessary. Consider if particular webpack plugins or loaders have memory leaks or unique memory usage characteristics.

Example: optimization and realContentHash

For instance, the `optimization` configuration for webpack has a `realContentHash` setting. This setting enables webpack to process assets again to generate accurate content hashes: https://webpack.js.org/configuration/optimization/#optimizationrealcontenthash

Webpack uses an internal hash calculation when this setting is disabled. This internal hash calculation may yield different hashes for identical content. Typically, the realContentHash setting is enabled. You can see how much memory is used by the process of generating that real content hash in the heap profile. The realContentHash pass consumes much more memory when there are more assets. When more files are part of the build process, the build-time and memory usage may increase. To reduce memory usage, one option is to turn this setting off and accept the drawback. Another option is to crack open the webpack “optimization” plugin code and investigate the possibility of updating the realContentHash algorithm to consume less memory, for example, by redesigning the algorithm or by calling out to an optimized C/C++/Rust/Go or operating system utility function.

Profiling with Node.js, CPU, and Heap Snapshots

Instead of using the plugins for webpack, you can use Node’s profiling tools to get a heap snapshot or a CPU snapshot. You can use this to profile any part of the build process.

As part of your React build process, you have to examine the performance of webpack and the performance of ESLint or TypeScript compiler if you’re using TypeScript or the test framework, such as Jest or Vitest. Each piece of the build-time process could become a performance issue. At runtime, you can use the profiling tools in Chrome Dev Tools.

Jest tests, for example, have had an issue with performance since October 2021, where the heap size became larger with each new Node version, indicating a memory leak. The thread about this problem has hundreds of comments from various investigations. It’s only been potentially solved recently. Sometimes, there is more than just a cost in time for slow build-time processes such as test runs; the cost is also in dollars if you are using GitHub Actions or another Continuous Integration tool to run the tests.

Example: Profiling a newly created React App

Let’s walk through an example of using these tools. I used Create React App (with TypeScript) to generate a new app and updated the webpack configuration to include the above-mentioned profiling tools. You can view the repo on GitHub here: https://github.com/rudolfolah/npm-webpack-profiling

Then, I created a baseline heap memory profile. The total memory used was 119 MB. By looking at the flame graph of the profile, it was possible to see that a big chunk of memory (52 MB) was used in the middle of the build process.

A few rows of the flame graph showed where the files were and were part of the ESLint webpack linter plugin. Checking the webpack configuration generated by create-react-app, an environment variable turned the ESLint check on or off. I turned it off and ran the build process to create another heap profile. This time, the total memory used was 54 MB. Quite an improvement!

Then, the source map generator used the most significant amount of memory. One thing to note here again is that you need to understand what the goals of the build process are; source maps can be helpful when debugging in a production environment and can be sent to tools like BugSnag or Sentry to have more informative tracebacks when an error occurs. Since this is an example to demonstrate profiling, I also turned that off. The total memory used was then 42 MB, an improvement that you can weigh against the usefulness of the source maps. So, we can live with a 54 MB build process with source maps.

When examining the Profiling plugin output, you can see that the disabled ESLint plugin and source map generator are no longer there.

What’s in the webpack Config?

Another plugin that’s useful for performance analysis is the webpack-config-dump-plugin. If your webpack configuration has conditionals and imports parts of it, it cannot be easy to trace which options and configurations were part of the webpack build process. The webpack-config-dump plugin will save the configuration with the values.

Here’s how I have been using it in the webpack configuration:

const { webpackConfigDumpPlugin } = require(“webpack-config-dump-plugin”);
new webpackConfigDumpPlugin({
  showFunctionNames: true,
  keepCircularReferences: true,
  includeFalseValues: true
})

Webpack Bundle Analysis

Tree-shaking with ESM modules

Through tree-shaking, webpack actively removes unused functions and code from its bundles. Suppose you use a library with ten functions but only import and call one. In that case, webpack will actively remove the nine unused functions from the created bundle. This process is valuable for the libraries you make, and it’s essential to determine if the libraries you use can undergo tree-shaking.

If you build libraries correctly, they can take advantage of tree-shaking. This issue might arise if the library doesn’t implement ES modules, lacks a “sideEffects” specification in its package.json, or has dependencies that don’t support tree-shaking. Some JavaScript libraries face this issue; although they support tree-shaking, their dependencies still need to. You can observe this outcome when you display the content of concatenated modules in the webpack bundle analyzer’s treemap.

Webpack Loaders

Wherever there is an import/require statement that uses a non-JavaScript file as part of the module path, webpack tries to find a loader that matches and then uses the loader instead of module resolution. For example, importing a .scss file would use the Sass CSS loader, which would transform the .scss into .css and return some code for the bundle. You can also achieve this with images and SVGs by incorporating them into the bundle’s static assets and copying them over.

It can be worth considering whether those loaders are still needed and if there are replacements for the part of the process that requires them.

Remove or Replace Packages

Some libraries and packages are relatively large for their features and business value.

Here’s an example. Datetime functions are essential. However, the most commonly used datetime library is moment.js, which will cause your webpack bundle size to grow by ~250 kb. Lighter-weight alternatives can be used instead, such as luxon, which will increase the bundle size by ~178 kb. Another alternative is date-fns, which also supports tree-shaking.

Understanding how much value you get out of the package and whether there are replacements for it can help reduce the bundle size.

Another option is to remove a package altogether. You can sometimes do this by rewriting the code to avoid the package dependency or to reuse some other existing code. In other cases, you can inline the functionality you need precisely, though this depends on the package license and how much effort you want to spend on incorporating the code.

Off-load Packages to CDNs using “externals”

CDNs (Content Distribution Networks) host the most popular JavaScript packages. You can load libraries like lodash or React from CDNs such as cdnjs or unpkg.

Your webpack configuration can flag those dependencies as “externals,” which means they will not be parsed or compiled into your bundle. There are a few advantages to this: CDNs are fast and near the edge where your customer is located; commonly used libraries on popular CDN URLs are likely already cached by the browser from visiting other websites; the webpack bundle becomes smaller.

Webpack supports multiple types of externals, from commonjs to umd to module to var (the default).

Split Packages into Sub-Packages, from Monolith to Mono-repo

To reduce the impact of a package on your bundle’s size, move it into a sub-package, especially for code parts that change infrequently. Combine this with a CDN and specify it as an external package to shrink the primary app bundle size. This approach also ensures that client web browsers retain the CDN packages in their browser cache on subsequent visits if unchanged. Alternatively, the new sub-package might enable tree-shaking, mainly if it is a utility library used by multiple apps, where each relies on a distinct set of functions.

Lazy-Loading with React

In React, you can use the “lazy()” method and Suspense to wrap a component, ensuring it loads only when needed. Webpack converts it into a separate chunk, reducing the main app bundle’s size. This approach speeds up the initial page load by preventing the loading of unneeded code.

An issue may arise when you import a component directly instead of using the “lazy()” method. Direct imports of some pages or subsections into another page can embed that chunk back into the main bundle. You should carefully analyze each file and component’s import declarations and dependencies. Analyzing the bundle and visualizing the connections between the chunks and assets will make it easier to identify if you are re-importing a piece of code and embedding it into multiple chunks or the main app bundle.

Setting Goals for Frontend Performance

Webpack allows you to set a performance budget for bundles and assets, which significantly assists in the early stages of developing a new app and adding features. If your app surpasses this performance budget, consider implementing one of the previously mentioned options or expand the performance budget. When engineering frontend performance, whether during build-time, at runtime on initial page load, or during user interactions, you must base decisions on business value, engineering value, and customer experience.

Depending on what kind of app you’re building and usage patterns, a vast bundle can deliver extraordinarily great value, or several small bundles with externals/CDN-hosted libraries and lazy-loading could provide the same or better business value and be of excellent engineering value during the development process. Setting these goals, understanding the value, and respecting your team’s expectations is highly important.

Data Pipeline for Product Analytics

There are three parts to a data pipeline for analytics: ingestion, processing/transformation, and interpretation. For analytics, many of the tools encapsulate this data pipeline. For ingestion of data, you need data sources. Google Analytics GameAnalytics or Mixpanel SDKs and implement event tracking in various places (web server, web app, mobile app, game, command-line tool). As the data is gathered, it is processed in a variety of ways; for instance, Mixpanel shows the cohorts and Google Analytics parses the web browser versions used to access a website. The interpretation step is all the dashboards, charts, and graphs that the analytics tools provide.

The data pipeline becomes a feedback loop: the data points collected become part of evaluating new and existing features for the product roadmap; new features need new event tracking created so the feature can be evaluated.

Adding a layer of sophistication, your data pipeline grows, and additional customization in the processing/transformation step becomes necessary, or the skillset and workflow of your data science team are based on generating and interpreting SQL reports.

Suppose you are happy with some of the processing and transformations. In that case, you can keep using whatever analytics tool you have and export the data to ingest it into a system that supports SQL queries.

Data Pipeline for AI and Machine Learning

For AI and machine learning, the data pipeline consists of four parts: ingestion, processing/transformation, training, and prediction. For AI/ML, ingestion consists of two key pieces: the data used for the initial training set and the feedback used for further training. The processing/transformation part transforms the events into training data. Then, the model is trained, tested, and revised in an iterative process. When it is deployed, it will respond with predictions. It can be trained further with feedback from customers, users, or other datasets.

Search engines and search results ranking are excellent examples of the pipeline. The data ingested are web pages; the processing/transformation extracts the title and keywords of the web pages and the number of clicks. The training uses that dataset, and then predictions are made. The feedback for search results can be measured in increased clicks for better results.

A crucial difference between the data pipeline for analytics and AI/ML is that the processing and transformation stages can include data labeling by humans or other synthetic data for training. For example, suppose you are training a programming ML model based on Python source code. In that case, you can supplement actual code bases with synthetic code that replaces variable names. Another example is image labeling or search results labeling, categorizing the image containing cats or dogs, and classifying the search results if they are irrelevant.

Data Pipeline for LLMs (Large Language Models), GPT-3, GPT-4, ChatGPT

The data pipeline for large language models or any pre-trained model such as GPT-3.5 and GPT-4 has the following steps: ingestion, processing / transformation, and evaluation. The processing step is to set up and recall context and the prompt for the large language model. The training and prediction steps that other AI/ML data pipelines have is skipped since large language models function differently. Fine-tuning does not change this either.

For example, if we are building a financial statements classification AI model using GPT-4 or ChatGPT, we would ingest the financial statements and transform them into a table with additional context such as the list of categories and then get a result. The result is then evaluated and the prompt is updated or continued.

Since pre-trained LLMs are accessed through external APIs, monitoring the response times and availability of them and monitoring any changes made to models is highly important. LLMs that are not built in-house can drastically change between models and prompts and contexts need to be evaluated constantly.

Data Pipelines are Everywhere

The most common data pipeline that software engineers will encounter is error tracking. Whenever an error occurs in their application, the error with contextual data and a stack trace are sent to an error monitoring tool. That tool, such as Sentry or BugSnag, will process and transform that data so that it can be interpreted. The tool encapsulates the data pipeline and the main focus becomes interpreting the data.

Another common data pipeline for software engineers is observability and logging through Splunk or Honeycomb or Datadog. With auto-instrumentation, the baseline of data ingested is great and the initial focus can be on interpreting data. However, more data will be needed to monitor the health of the system, that data could be annotations of when there was a release or deploy, or additional context about a request or database query. The focus is an iterative process of interpreting data and updating the ingested data to better understand and monitor the system.

For startups and public companies and businesses everywhere, the most important data pipeline is the revenue and expenses. The data can come from bank accounts or PayPal or Stripe, it’s processed and transformed into an income statement in QuickBooks or Xero or another accounting tool and then interpreted at different levels of time (daily, monthly, yearly). When the financial data pipeline is working smoothly, interpreting the data is the primary focus. When the pipeline is being set up, the main focus is on the ingestion of the data. Sometimes there can be issues in the processing/transformation step, for example double-counting or excluding items that should have been included. The tools that are used for this pipeline all have APIs so that they can at the beginning of a customized data pipeline, for instance include flagging large expenses for review or sharing large sales for celebrations.

Building a Customized Data Pipeline

At a high-level, to build a data pipeline you will need a way to receive data, an ETL (extract, transform, load) to process/transform it, and then a way to view the results for interpretation.

The simplest customized data pipeline consists of a web server and a database. Using Ruby on Rails or Django, you can ingest data with one endpoint, transform it in a background process, and then store the data in a database. On another endpoint, you can display charts based on that data. Make sure to keep an eye on how much data is generated and consider retaining it for a short period of time. Some other options for limiting the amount of data generated are: aggregating it as part of the transformation step, or archiving it to cold storage (such as S3 or Glacier).

With this as a starting point, you can improve each part of the pipeline.

For instance, you can improve the data ingestion and make it scalable by running it as a separate web service with auto-scaling. More users means more requests to the ingestion endpoint means more writes to the database.

To improve the interpretation part of the pipeline, you can upgrade the database with more read replicas and switch your web app to use those for reading the data and displaying it. At a certain scale, you can batch those writes in asynchronous background distributed tasks. This can improve data ingestion and improve the processing/transformation parts of the pipeline.

If you’re interpreting data on a weekly basis, you can improve the processing/transformation part of the pipeline by aggregating the data for each week ahead of time, acting as an index for the data interpretation.

At the other end of the scale, at enterprise scale, you will need a data warehouse and you will have many sources of data being ingested and transformed. Tools such as Looker and Tableau can use the data warehouse and display charts for interpretation of the data.

What is a Smart Contract? What can they be used for?

A smart contract is a program that runs on a blockchain. Smart contracts can have custom logic and their own method of keeping track of who owns what.

They are used for NFTs and decentralized financial protocols. All collectible NFTs on Ethereum and on various Cosmos-based blockchains are kept track of within a smart contract.

DeFi protocols use smart contracts for keeping track of loan collateral, wrapped tokens, swaps, trades, decentralized exchanges and more. Smart contracts are executed on nodes within the blockchain and can be queried for data and information.

What is Cosmos?

Cosmos is a blockchain that makes building new chains easier than before, also known as an internet of blockchains. Cosmos functions as a foundation that can be used to build application-specific blockchains that are interoperable.

The primary token for each chain is used to pay gas fees (gas keeps track of the amount of resources used during execution of a transaction). The cryptocurrency token for Cosmos is ATOM, and each application-specific chain has their own tokens such as STARS for Stargaze, JUNO for Juno, OSMO for Osmosis, and SCRT for Secret Network.

Cosmos supports smart contracts through CosmWasm. You can write smart contracts using the Rust programming language for Cosmos-based blockchains that support CosmWasm.

Which Cosmos-based blockchains support CosmWasm?

The following Cosmos-based blockchains support running CosmWasm smart contracts:

• Cudos, a cloud computing blockchain
• Juno, a chain for smart contracts and app development
• Osmosis, a decentralized exchange, recently enabled CosmWasm support
• Secret Network, a privacy-first chain, supports a modified version of CosmWasm

This list is based on the documentation for the chains and based on the Cosmos chain registry repo on GitHub (where the schema will contain cosmwasm_enabled).

How do you write smart contracts on Cosmos? How do you write an NFT or tokens contract?

You can write smart contracts for Cosmos-based blockchains using the Rust programming language and the CosmWasm SDK and libraries. Rust is a typesafe and performant language that can be used to write any sort of software from web servers to game engines to smart contracts. It improves the reliability of programs and can be used in resource-constrained environments where efficiency is a priority.

Learn more about Rust

Here are the base libraries you will need to get started with writing CosmWasm smart contracts:

• cosmwasm-std contains standard utility functions for working with wallet addresses and coins
• cosmwasm-storage contains code for storing data as maps/dictionaries/hashes and as individual values
• cosmwasm-schema helps generate the JSON schema for interacting with the smart contract

If you want to write an NFT smart contract, your contract’s interface will need to adhere to the CW721 interface. You can use the cw721-base code to build your own NFTs smart contract.

NFTs are non-fungible tokens, each NFT has a unique identifier and belongs to one owner. Some popular examples of NFTs are Cryptokitties, Doodles, CryptoPunks, BAYC (Bored Ape Yacht Club), and Loot. Your NFT smart contract can include code that calculates royalties whenever an NFT is bought or sold, or it can include additional details about the NFT such as the traits or image data to use. A really creative developer can write Rust code that evolves the NFT over time or based on certain interactions. For instance, every time the NFT is traded, one of the traits could have a new value or the image data could be modified.

If you would like to create fungible tokens, and issue your own cryptocurrency, you can do that by writing a smart contract that adheres to the CW20 specification.

When you write your smart contract, you can re-use code from other smart contracts by adding their crates to your Rust project. As mentioned above, there are standard libraries that you can use for CosmWasm, and there are additional packages that can be used as a base for CW20 fungible tokens and CW721 Non-Fungible Tokens.

During a smart contract consulting project, we implemented a way for NFTs of one type to be exchanged for another type of NFT that was randomly chosen from the set of remaining unowned NFTs. We had to adhere to the CW721 NFT contract interface to make it work.

How does a Cosmos smart contract interact with another smart contract?

Interactions between smart contracts allow composability of contracts, where you can build on top of NFT collections and DeFi protocols.

As part of an NFT consultation and implementation project, we had to interact with other smart contracts. One of the interactions was with a CW721 NFT smart contract in order to verify ownership of an NFT that would be exchanged. Another interaction was with a randomness oracle smart contract, LoTerra, which would return a random number. As part of the smart contract execution, the query messages would be sent out to the other smart contracts, return with responses and continue on the happy path to update the ownership of the NFTs.

To interact with another contract, you can execute a message, which can change state and update the storage in the other contract, or you can query that contract for information. In the Rust smart contract code, this is implemented by creating new message structs that are converted and passed along to the CosmWasm querier.

This code shows how to build a query message that queries another smart contract:

fn is_airdrop_eligible(deps: Deps, owner: String) -> bool {
    let msg = hackathon_nfts::msg::QueryMsg::Balance { owner, token_id: "example".to_string() };
    let wasm = WasmQuery::Smart {
        contract_addr: HACKATHON_NFTS_CONTRACT_ADDR.to_string(),
        msg: to_binary(&msg).unwrap(),
    };
    let res: StdResult<hackathon_nfts::msg::BalanceResponse> = deps.querier.query(&wasm.into());
    match res {
        Ok(res) => res.balance.u128() == 1u128,
        _ => false,
    }
}

Unit Testing and Mocks and Test Data for CosmWasm smart contracts

What’s great about Rust is that you can add unit tests with mocks to thoroughly test various query and execute responses from another smart contract.

Here is an example of a mock querier used in a unit test where the response is mocked with fake data for testing:

#[derive(Clone, Default, Serialize)]
pub struct HackathonNftsBalanceResponse {
    pub balance: Uint128,
}

impl HackathonNftsBalanceResponse {
    pub fn new(balance: Uint128) -> Self {
        HackathonNftsBalanceResponse { balance }
    }
}

// ...

impl WasmMockQuerier {
    pub fn handle_query(&self, request: &QueryRequest<TerraQueryWrapper>) -> QuerierResult {
        match &request {
            QueryRequest::Wasm(WasmQuery::Smart { contract_addr, msg }) => {
                if contract_addr == &"terra1utd7hcq00p0grm08uyg23mdm782726y3euk8pq".to_string() {
                    let msg_hackathon_nfts_balance = HackathonNftsBalanceResponse {
                        balance: Uint128::from(1u128)
                    };
                    return SystemResult::Ok(ContractResult::from(to_binary(&msg_hackathon_nfts_balance)));
                }
                panic!("DO NOT ENTER HERE");
            },
            _ => self.base.handle_query(request),
        }
    }

    pub fn new(base: MockQuerier<TerraQueryWrapper>) -> Self {
        WasmMockQuerier { base }
    }
}

Through unit testing, it’s possible to test a large variety of different parameters and thanks to Rust’s speed and performance, you can write more tests and create more test data for more thorough test scenarios.

Here is an example of a unit test for a Cosmos smart contract:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::mock_querier::mock_dependencies_custom;
    use cosmwasm_std::testing::{mock_dependencies, mock_info, MOCK_CONTRACT_ADDR};
    use cosmwasm_std::{
        coins, from_binary, Addr, BlockInfo, ContractInfo,
    };

    // ...

    #[test]
    fn mint_pizza_and_pie() {
        let mut deps = mock_dependencies_custom(&[]);
        let res = instantiate(
            deps.as_mut(),
            mock_env(0),
            mock_info("creator_address", &coins(1000, "earth")),
            InstantiateMsg {},
        ).unwrap();
        assert_eq!(0, res.messages.len());

        // regular mint before aidrop is claim gives two pizzas
        let res = execute(
            deps.as_mut(),
            mock_env(1),
            mock_info(
                "player_address",
                &[Coin {
                    amount: Uint128::from(1_000_000u128),
                    denom: "uusd".to_string(),
                }],
            ),
            ExecuteMsg::MintPizza {},
        ).unwrap();
        assert_eq!(0, res.messages.len());

        let res = query(
            deps.as_ref(),
            mock_env(2),
            QueryMsg::Inventory {
                address: "player_address".to_string(),
            },
        ).unwrap();
        let inventory: InventoryResponse = from_binary(&res).unwrap();
        println!("{:?}", inventory);
        assert_eq!(2, inventory.pizzas.len(), "there should be two pizzas from mint + airdrop");
        assert_eq!(0, inventory.pies.len(), "there should be no pies, no pizzas have been combined yet");

        // ...
    }
}

One great benefit of CosmWasm and Cosmos-based blockchains is that you can run an array of tests within a local development testnet to ensure that interactions with your smart contract are working correctly in terms of transactions, wallet balances and storage updates. Local development testnets provide a way to test for various scenarios. It’s recommended to test thoroughly on a testnet and, especially for DeFi protocols, to hire an auditing firm to audit the smart contract code.

Here is an example of how to run JavaScript code that executes transactions on a local testnet:

const coins = { uluna: 1_000_000, uusd: 1_000_000 };
console.log('sending coins from multiple senders to one receiving wallet');
for (let wallet of wallets) {
    if (wallet.address === wallets[0].address) {
        continue;
    }

    const send = new MsgSend(
        wallet.address,
        wallets[0].address,
        coins
    );
    try {
        const tx = await terraWalletFromMnemonic(wallet.mnemonic).createAndSignTx({
            msgs: [send],
            memo: 'terrajs-test-harness',
            fee: new Fee(2_000_000, { uusd: 10_000_000 })
        });
        const result = await terra.tx.broadcast(tx);
        await displayTxInfo(result.txhash);
    } catch (e) {
        console.log(e);
    }
}

Machine learning and deep learning techniques have become popular topics over the past five years. Machine learning is being applied to many problems and we are seeing some interesting and effective results. Machine learning is a way to process and analyze big data and make predictions about future data. For instance, you can analyze millions of emails to identify which ones are spam and which ones are not, and you can analyze this based on hundreds or thousands of features or signals.

To get you started with machine learning, you could start with TensorFlow, a machine learning library open-sourced by Google in 2015. Alternatively, you could start with scikit-learn, or perhaps another library.

However if you want to get started quickly, you should consider trying out Ludwig.

The code for this article is available on Github: https://github.com/modernlearner/machine-learning-with-ludwig

Ludwig is a free/open source machine learning library created by the software engineers are Uber. It combines TensorFlow, SciKit-Learn and other Python packages into an easy-to-use package where you can load data sets from CSV files, train your machine learning model, and then apply it to new data. It can visualize the performance of the machine learning models you build and show you comparisons. It can save a snapshot of your model so that you do not have to re-train it from scratch and just need to feed it additional data. It is by far the simplest way to get started in applying machine learning.

Assumed Knowledge

For this article, we assume that you know how to write code in Python and how to run Python programs. We are using Python 3.8 in this article.

I am also assuming that you have a small amount of knowledge about machine learning, such as what a neural network is.

Installing Python and Ludwig

Let’s begin by installing virtual env and creating the environment:

sudo pip install virtualenv virtualenv –python=python3.8 env

Then let’s install the dependencies:

source env/bin/activate pip install -r requirements.txt

Install English-specific encodings for spacy language processing:

python -m spacy download en

Installing Ludwig

The dependencies list looks like this:

ludwig[text,viz]==0.3.1

This will install Ludwig, TensorFlow, SciKit-Learn, Pandas, Numpy. It will also install SpaCy for text and natural language processing. Additionally, it will install visualization tools like matplotlib and seaborn.

Starting Code

In the Python files that we create, you should start with this chunk of code:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
import os.path

import pandas as pd
from ludwig.api import LudwigModel

The first part will disable the output of some logging from TensorFlow. This can be a bit noisy. The environment variable TF_CPP_MIN_LOG_LEVEL controls the level of logging. Its value can range from 0 to 4, where the highest number 4 will show more logs (info, warning, error, fatals) and the lowest number 0 will show no logging information.

The other half of that code will import the necessary modules to be able to use Ludwig.

And with that we’re ready to write our first machine learning program.

Basic Prediction with Ludwig

We are going to build a machine learning model and train it to identify which emails are spam and which are not. Let’s take a look at the data set we are going to use to train the model.

The Dataset: Email Spam

Our data set is a CSV (comma-separated values) file with three columns:

• subject
• content
• spam

All three columns are strings of text. However, as we will see in the next section, they are processed differently when it comes to defining our machine learning model.

Here is a sample of the data (you can check out the code repo on Github to see all of the data):

"cruise winner!","Hello! You just won a cruise!","spam"
"BANK WARNING","This is your bank, warning you that someone has tried to access your account","spam"
"RE: AI meeting","The agenda for our company meeting is AI and machine learning. Please be on time. Thanks!","not_spam"

As you can see from this sample, the last column in some rows of data is “spam” and in other rows it is “not_spam”.

Important Notes about the Data Set

One important thing to note here is that I, the author, of this article, have classified the data rows as either spam or not spam.

It is up to you to double-check the data set you are using for training matches is correct.

It is up to you to analyze the data set and see if there are biases and make sure to correct for them.

Another important thing: this sample of data is not enough to properly train the machine learning model, there are not enough data points. The small data set in the code repo has some more data points and it is just barely enough to train the model. It’s a good starting point and we are using a small data set so we can understand roughly what is happening within our model. Remember that companies such as Google, Facebook and Uber train their machine learning models on data sets where there are millions of data points. To train this model further you can use “online” training which is provided by the Ludwig method: https://ludwig-ai.github.io/ludwig-docs/api/LudwigModel/#train_online

Ludwig Configuration: Input and Output

So we have a data set with three columns. Our machine learning model needs input features to “learn” from, to be trained on, and it needs output features that we would like to predict or calculate. The “features” of a data set describe the shape of the data.

Input Features

The input features for the email data set are going to be the “subject” and the “content” columns.

With Ludwig, you can start with this very simple definition of model input features:

input_features = [
    {
        'name': 'subject',
        'type': 'text',
    },
    {
        'name': 'content',
        'type': 'text',
    },
]

Ludwig provides default values for the encoding of the columns that are used as input features and it provides default configurations and parameters for the underlying machine learning neural network to use.

You can override the default configuration of the input features and specify your own configuration:

input_features = [
    {
        'name': 'subject',
        'type': 'text',
        'preprocessing': {
            'lowercase': True,
        },
    },
    {
        'name': 'content',
        'type': 'text',
        'preprocessing': {
            'lowercase': True,
        },
        'encoder': 'stacked_parallel_cnn',
        'reduce_output': 'sum',
        'activation': 'relu',
    },
]

Preprocessing Data

The preprocessing step we are using for the subject and content fields is “lowercase”. This means an email with the subject “Hello WORLD” will look the same as “hello world” and this could affect your training results and the performance of the machine learning model.

Other preprocessing options for text features include controlling how characters are mapped, how words and characters are tokenized, and the number of common words to consider. You can read more about them here: https://ludwig-ai.github.io/ludwig-docs/user_guide/#text-features-preprocessing

Encoding Data into a Neural Net

The “text” type of input feature uses the spaCy natural language processing library to tokenize the text. For example, “hello world” in an email subject will be transformed into a list of two word tokens, “hello” and “world”. The machine learning model will keep track of these tokens and how often they are used across all the data points.

For the “subject” input feature we’re using the defaults that Ludwig supplies for the “text” feature type.

However, for the “content” input feature, we are overriding the default encoder, we are overriding how the output is reduced in the encoder and how the activation of neural net nodes functions. The encoder specified is stacked_parallel_cnn. This is a convolutional neural network that has multiple layers that are stacked. Within each stacked layer there are parallel layers. As inputs are processed they go through these layers which combine and activate in a particular way.

You can read more about it in the Ludwig User Guide: https://uber.github.io/ludwig/user_guide/#stacked-parallel-cnn-encoder

Remember, Ludwig gives you easier access to creating machine learning models and creating a pipeline for data and model training.

It is up to you to understand how and why to use specific encoders and specific options for the encoders. With respect to the spam emails you may get better results with a different encoder entirely than the one used here.

Output Features

After defining the input features, we need to define the output features:

output_features = [
    {
        'name': 'spam',
        'type': 'category',
    }
]

We want to know which category the given input features amount to. The category is either spam or not_spam.

After we train the model on the dataset, we will be making a prediction for another dataset that only includes the two input features and we expect the prediction of the output feature category to match our expectation.

Training and Testing a Ludwig Machine Learning Model

Now that we have the input and output features of the model, we can define the model configuration:

base_model_definition = {
    'input_features': input_features,
    'output_features': output_features,
}

We need a function that will load an existing model or create a new model using that model configuration:

def load_or_create_model(model_dir, model_config, **model_kwargs):
    if os.path.exists(model_dir):
        print('Loading the model: {}'.format(model_dir))
        return (LudwigModel.load(model_dir, **model_kwargs), True)
    else:
        print('Defining the model')
        return (LudwigModel(model_config, **model_kwargs), False)

First we check if the model already has been trained and saved to a directory. If that’s the case, we simply use LudwigModel.load to load up the pre-trained model. However, if that is not the case, we need to create a new LudwigModel using the model configuration.

Now let’s use this function to try to create/load the model:

base_model, base_model_loaded = load_or_create_model(
    'trained/basic', base_model_definition, gpus=[0], gpu_memory_limit=2000
)
if not base_model_loaded:
    print('Training the model')
    base_model.train(
        training_set='./datasets/spam_train.csv',
        test_set='./datasets/spam_test.csv',
        skip_save_processed_input=True,
    )
    base_model.save('trained/basic')

Notice that we supplied two keyword arguments, gpus and gpu_memory_limit, if you do not have a GPU you can remove those. If you have one or more GPUs or more GPU memory available, you can update the list of GPU IDs and increase the memory limit. When running just on CPU with no GPU, it can take much longer to train a model. We want a faster feedback loop if we are adjusting multiple parts of the model and trying to compare a group of machine learning models.

Back to the code! If the model is newly created, we’re going to use the train method with a training set and a test set. The training set is the data that is processed and used to train the model. The test set is another set of data (usually a subset of the training) that is used to check the machine learning model for accuracy. If the model is trained well, it will have a higher accuracy rate on the test data set.

Additionally, we skip over saving any processed input, we want the input to be processed every time. It’s just one less thing to delete if you’re changing the model and re-training a few times).

Then we save the model after it has been trained into a directory.

Now we’re going to evaluate the model against the test dataset:

stats = base_model.evaluate(
    dataset='./datasets/spam_test.csv'
)[0]
print(stats)

When we print out the stats, we shall see the accuracy of the model. When you run the code, you will see that the performance is at a certain level. To improve that performance you can delete the trained model from its directory and then update the model configuration and re-run the program.

Creating a Second Model

Now let’s create a second model that is adjusted and compare its performance to the first model.

We begin by copying the first model’s configuration:

print('Creating a 2nd model classifier with some adjustments')
other_model_definition = base_model_definition.copy()

Then we override the input features:

other_model_definition['input_features'] = [
    {
        'name': 'subject',
        'type': 'text',
        'preprocessing': {
            'lowercase': True,
        },
    },
    {
        'name': 'content',
        'type': 'text',
        'preprocessing': {
            'lowercase': True,
        },
        'encoder': 'bert', # See: https://ludwig-ai.github.io/ludwig-docs/user_guide/#bert-encoder
        'reduce_output': 'avg',
        'activation': 'tanh', # Activation used for *all* layers.
        'num_filters': 64,
        'stacked_layers': [
            [
                { 'filter_size': 2 },
                { 'filter_size': 3 },
                { 'filter_size': 4 },
                { 'filter_size': 5 }
            ],
            [
                { 'filter_size': 2 },
                { 'filter_size': 3 },
                { 'filter_size': 4 },
                { 'filter_size': 5 }
            ],
            [
                { 'filter_size': 2 },
                { 'filter_size': 3 },
                { 'filter_size': 4 },
                { 'filter_size': 5 }
            ]
        ],
    },
]

There are a lot of changes to unpack here.

Differences Between the First Model and the Second Model

Encoders

The first difference in our models is that this second model uses the BERT encoder when encoding the content input feature. This means that the text is processed and encoded using BERT (Bidirectional Encoder Representations from Transformers). This encoder is pre-trained to work on English text. This is important because if you want to classify spam in other languages, you will either need to use a BERT model that’s pre-trained on your target language or you will need to train it yourself.

If you want to change the language, add pretrained_model_name_or_path and set it to one of the values here: https://huggingface.co/transformers/pretrained_models.html

Later on you can change the encoder from BERT to GPT or T5 or ELECTRA. There are many text encoders available with Ludwig.

Reduction of the Neural Net Output

The next difference is that we’re are reducing the output of the neural network from 'sum' to 'avg'. This reduces the output tensor using the average value instead of the sum, for example if we’re seeing 0.5 and 0.5, we’d get 1.0 in the first model and 0.5 in the second model.

Other ways to reduce the output include 'min', 'max', 'concat', 'last' and 'null'. Again, you will need to read further about each of these to understand which is the best to use.

This is one of the important parts of machine learning. You have to understand the problem you’re applying machine learning to and then you have to understand what tools are available in the toolbox and dig deep into the details to see if you can produce a more accurate machine learning model.

Neural Net Activation Method

The activation method is a function applied to the output. The first model uses the default which is the relu function. The second model uses tanh instead. “Relu” means “rectified linear unit” while “tanh” means “hyperbolic tangent”.

Here are charts of how these activation functions look:

You can find more activation functions in the TensorFlow (or Keras) documentation: https://keras.io/api/layers/activations/#layer-activation-functions

Number of Filters

The number of filters is also different between the two models. In the first model, we’re using the default which is 256, while in the second model we’re using far fewer filters, 64. The number of filters controls how many output channels there are.

If you want to learn more about filters and how they work in a neural network as part of machine learning, check out this article: https://machinelearningmastery.com/convolutional-layers-for-deep-learning-neural-networks/

Filter Size in Each Neural Network Layer

The filter size controls the width of the convolutional filter. The filter size is set within each layer of the neural network with different filter sizes in different layers.

Training the Second Model

Now let’s load or create the model and train it:

other_model, other_model_loaded = load_or_create_model(
    'trained/basic_other', other_model_definition, gpus=[0], gpu_memory_limit=2000
)
if not other_model_loaded:
    print('Training the model')
    other_model.train(
        training_set='./datasets/spam_train.csv',
        test_set='./datasets/spam_test.csv',
        skip_save_processed_input=True,
    )[0]['training']
    other_model.save('trained/basic_other')
other_stats = other_model.evaluate(
    dataset='./datasets/spam_test.csv'
)[0]

Comparing the Performance

Ludwig provides multiple visualization functions to show you graphs and charts of the performance and accuracy of the machine learning models. Remember that Ludwig is made so you can create many models and compare their performance easily to select the most promising model and adjust its parameters further.

Let’s see how well our first model for email spam detection works in comparison to our second model:

visualize.compare_performance(
    test_stats_per_model=[stats, other_stats],
    output_feature_name='spam',
    model_names=['Base Model', 'Other Model'],
)

We’re using the compare_performance function from Ludwig, you can read more about it here: https://ludwig-ai.github.io/ludwig-docs/user_guide/#compare_performance

Applying the Models and Using Them to Predict Spam Emails

We have two machine learning models that are trained and tested and we have compared their performance. Let’s use those models and predict some more spam emails.

Here’s a convenient way to run the predictions and print out the result:

def print_predictions(unpredicted_emails, model):
    prediction_result, output_directory = model.predict(dataset=unpredicted_emails)
    emails = unpredicted_emails.join(prediction_result)
    for index, row in emails.iterrows():
        print('{} ({:.6f}): {} / {}'.format(
            row.get('spam_predictions'),
            row.get('spam_probability'),
            row.get('subject')[0:30],
            row.get('content')[0:30],
        ))

We’re using the predict method on the model and this is what you would use to see if a new email is spam. If the model is incorrect, you can train it further by correcting the data and re-training model (or by using online training).

And now let’s run the two models and predict spam emails in the “unpredicted” CSV file:

unpredicted_emails = pd.read_csv('./datasets/spam_unpredicted.csv')
print('Prediction Results for Base Model')
print_predictions(unpredicted_emails, base_model)
print('Prediction Results for Other Model')
print_predictions(unpredicted_emails, other_model)

Here’s how to run the code:

$ source env/bin/activate
$ python basic.py

Then you will see the comparison between the models and then the prediction for each email and whether or not the model considers it to be spam.

Where To Go From Here

With this base of knowledge, you can do the following:

• Find new datasets and create new models for them: you could try recognizing digits on a larger dataset, or train a classifier for something other than spam emails.
• Tweak the models: iterate on the models and tune them in different ways to see which will get you the best performance.
• Learn TensorFlow directly: skip Ludwig and learn how to create a neural network using TensorFlow and learn how all the layers and configuration parameters work.
• Get the example running on a cloud-based GPU: use Lambda Labs or Google Colab to create a compute instance that includes a GPU for you to train and test the models on.

AngularJS

• How to Use Local Storage for Caching in AngularJS 1.6
• SiberianCMS: make amazing mobile AngularJS apps
• A Primer on AngularJS Services
• Unit Testing in AngularJS
• AngularJS Books
• Top 15 Awesome Tutorials to Learn AngularJS
• Deconstructing AngularJS Injector

How to Use Local Storage for Caching in AngularJS 1.6

Window Local Storage is an HTML5 web api that can be used to store data in the browser. The data is stored in a bucket for the current domain name (click here to view the Storage API).

An awesome developer created the angular-local-storage module for AngularJS 1.6 which gives you a nice API to work with in AngularJS for local storage.

What’s cool about is that you can listen for notifications whenever the local storage has been changed (with setItem or removeItem). It also lets you set the storage to use local storage or session storage. Session storage is great for quick caching while local storage is more long-term.

AngularJS API for Local Storage

The API for angular-local-storage is easy to use:

• isSupported – whether the browser supports local storage
• setPrefix – prefix used when setting keys
• getStorageType – the type of storage used
• set – set a value
• get – get a value
• keys – the list of keys that have been set in local storage (very very convenient!)
• remove – remove a value
• clearAll – remove all values set
• bind – binds a value from local storage to a variable in the $scope

The most important API functions for angular local storage are: set, get, and bind.

Here’s an example of how it’s used:

var myApp = angular.module('myApp', ['angular-local-storage']);
myApp.controller('HomeCtrl', ['$scope', 'localStorageService', 'someApiService', function($scope, localStorageService, someApiService) {
  
  function $onInit() {
    console.log('the keys! ' + localStorageService.join(', '));
    console.log('storing a value from an API call');
    someApiService.fetchValues(function(values) {
      localStorageService.set('valuesFetched', values);
    });
    localStorageService.set('formSubmitted', false);
  }

  function submitForm() {
    if (localStorageService.get('formSubmitted')) {
      console.log('form already submitted!');
    } else {
      localStorageService.set('formSubmitted', true);
    }
  }
}]);

In the example you can see that angular local storage is used to fetch values when the controller HomeCtrl is initialized with $onInit. After the value is fetched, we use the localStorage service to set the initial value for formSubmitted.

When the form is submitted through the submitForm method, we check the value of formSubmitted through localStorage and then set its value. The idea is to prevent resubmission of the form. Of course this is just for the frontend and it’s a good idea to check for resubmission on the server backend.

Angular local storage is simple to use and highly valuable as a cache for AngularJS web apps.

The State of Angular 2/4/5+

After the last year of keeping up with Angular 2/4/5+ development, it seems that they’re still gearing up. In my eyes, they have taken on a lot of work and have shifted the ground underneath developers quite a bit. The full adoption and support for TypeScript is welcome news, but there are still many development shops that will use JavaScript and ES6 with Babel. The documentation and examples do not seem to support JavaScript as much as they support TypeScript. This is also true for Dart, which is used by Google for one of the larger Angular projects, but even they cannot count on full support from the Angular team.

There’s also the rapid rate of change. Angular has an upgrade path however it is not fun to upgrade dependencies and change code every few months because the API of the framework has changed.

The state of Angular 2/4/5+ at this point is similar to the state of the Django web framework before it was version 1.0. Before 1.0 of Django, every single week seemed to bring new changes in the codebase. It was fun to be on the cutting edge and the latest and greatest but Django before 1.0 was not ready for production. This cycle is happening to Angular 2/4/5+. If you want to be on the cutting edge, use Angular and be careful when deploying and maintaining a production app.

When should you consider using Angular 2/4/5+?

If you are starting a new project, you should consider Angular 2/4/5+

If you want to develop a mobile app using Angular, use NativeScript with Angular 2./4/5+

If your web development team has at least 3 developers, consider using Angular 2/4/5+

When should you consider using AngularJS 1.6?

If you have an older AngularJS 1.x project, consider upgrading it to AngularJS 1.6 (don’t attempt a rewrite in Angular 2 until you do that!)

If your web development team has 1 to 3 developers, consider AngularJS 1.6 because you will find a lot of modules that have been production-ready and are well-tested. You will be able to ship the project into production and support it for years to come. As I pointed in the last Learning AngularJS issue about Siberian CMS, they are using AngularJS 1.3 for mobile apps and it still works just fine for them.

Thanks for reading!

Thanks for reading! If you’re still using AngularJS 1.6, don’t worry you are not alone, there are thousands of projects out there also using it and the modules are well-tested and production-ready! If you ever wanted to use local storage in AngularJS, now here’s your chance with angular-local-storage.

SiberianCMS: make amazing mobile AngularJS apps

SiberianCMS, a mobile AngularJS CMS, is a content management system. Much like Drupal or WordPress it is easy to manage a lot of content and to produce a mobile app.

Its most unique feature is that it makes mobile apps. The code will compile into a native Android or iOS/iPhone app with all of the content managed by the SiberianCMS backend. If you ever wanted to quickly create a mobile app, this might be the tool for you. I’m considering using it to lower the effort and cost required to create a mobile shopping app and to use it for hackathons.

Mobile AngularJS

As a CMS, SiberianCMS is very easy to use. With SiberianCMS, you can create new forms and pages with a GUI editor, similar to Drupal. It supports custom RSS feeds for adding content as well.

SiberianCMS is very extensible, there are a lot of great plugins/modules for it. Prices for the plugins range from free to over 150 euros. It’s possible for you to create your own plugins and make some money that way.

The plugins/modules for SiberianCMS include:

• chat/messaging
• calendar of events
• classifieds
• Uber-like taxi-ordering/navigation system
• maps and directions
• quizzes and surveys

The cool thing is that these modules are written in JavaScript and as AngularJS components and modules. You can hack on your own module today!

The code for SiberianCMS is open source so you are free to download it and create your own plugins for it. It uses the Open Software License, which is also used by the highly popular ecommerce engine, Magento.

Stable Frontend Mobile/Web Development Is Possible!

What I like, aside from SiberianCMS itself, is that you can build a stable system on AngularJS 1.x. You don’t have to be subjected to the cycle of frontend web framework churn. How many javascript frameworks exist now anyway?

You are able to build solid products for your startup or community or for yourself that can last for years by sticking to one version of AngularJS. For new projects, the choice isn’t between using Angular 2/4+ or AngularJS 1.6. You also have the option to use AngularJS 1.3 or 1.5 or whatever you’re comfortable with. But you should always upgrade.

However. if you want to maintain stability, AngularJS will help you achieve that. I’ve been a big advocate of Angular 2/4+ lately but it is still fun to work with AngularJS 1.x. Also, for new projects I still consider AngularJS 1.x a practical option.

I recommend you check out SiberianCMS for yourself. See how quickly you can prototype and launch a mobile AngularJS app! Maybe try hacking around their source code too.

Upcoming Angular Conferences

Here are a few conferences that are coming up:

• September 25 to 27, Angular Summit in Boston MA
• November 7 and 8, Angular Connect in London UK

A Primer on AngularJS Services

The article is by Michael Herman it’s a very nice short article on how to use AngularJS services.

The article talks about what a service is and how to create one. It’s very well done and the code is easy to follow and understand.

When you’re first starting with AngularJS it can be tempting to put a lot of functionality into a controller, but you should be separating concerns and moving some of that functionality into separate services that the controller uses. Even for small apps, it’s good to get into the habit of creating separate services.

As Michael says,

If your controller is handling more than just defining the scope or initial state of your app, connecting your models and views, then it’s are probably doing too much.”

The article is a prime example of great documentation of a core AngularJS feature.

Read the article

See a demo of the code in action.

Unit Testing in AngularJS

The modern way of development is to build and deliver web apps that include automated unit test suites. These automated tests exercise your web app to make sure that it works and doesn’t have any bugs. Tests give you the confident to say to your client or project manager or customer, yes we have the right features and yes they all work.

Test-driven development is a set of techniques that any software engineer can follow, which encourages simple design and test suites that inspire confidence…following these two simple rules can lead us to work much more closely to our potential.

• Write a failing automated test before you write any code.
• Remove duplication.

Kent Beck, “Test Driven Development”

Unit Tests

Unit testing is essential and of course, AngularJS lets you do some unit testing. In fact, it was built from the ground-up to co-operate with unit testing by using dependency injection.

One of the better tutorials on fully testing your AngularJS app is Full-Spectrum Testing With AngularJS and Karma.

To begin unit testing AngularJS, you need the following:

• Karma test runner
• Jasmine or Mocha/Chai test frameworks
• Sinon for mocking/spies/stubs

The Karma test runner will run whatever tests you have. The Jasmine test framework lets you define tests and has assertions (such as myValue.is.expected.toBe(3)). Personally, I prefer the Mocha framework since it only defines tests, and lets you use whatever assertion library you want to use such as Chai, it also has lots of nice features.

A classic on unit testing is Test Driven Development: By Example, it’s by Kent Beck who was one of the founders of the Extreme Programming movement.

Mocking Dependencies for Integration and Unit Testing AngularJS

Sometimes when you’re writing a test, you want to make sure that your service or provider is making the right calls to another service. The way to do this is with mocks. Mocks are fake objects that keep track of which methods were called on them.

Mocks are not essential for writing tests, but there are cases where they do help. When you’re writing a controller that connects to your web app’s REST API, you want to make sure that it calls methodX and methodY. By creating a mock service and registering it with AngularJS, you can write a unit test that makes sure those two methods have been called. When using external 3rd party REST APIs, you may want to mock the $httpBackend to make sure the right calls are being made and the correct responses are returned.

Sinon is valuable for defining mocks. It lets you create method stubs and it lets you spy on method calls.

AngularJS Modules:

• angularjs-nvd3-directives: directives for creating charts and graphs using nvd3.js or d3.js
• angular-file-upload: directive for handling file uploads using HTML5 when available
• angular-lorem-image: directive for placeholder ithat uses lorempixel service to generate images

News:

• Good Practices To Build Your AngularJS App: nice overview of AngularJS with good info on using scopes.
• AngularJS and scope.$apply: great explanation of how the digest cycle works and how to update data bindings.

AngularJS Meetups:

• Boston, MA
• Mountain View, CA
• New York City, NY
• Toronto, Canada
• London, UK
• AngularJS Conferences

AngularJS Books

Today we’re covering some AngularJS books that can get you started on your path to learning how to design single-page web applications using AngularJS. While the code examples in books can sometimes become outdated, books are still valuable because they go over and explain the major concepts (directives, services, factories, unit testing, etc.) that you need when developing an AngularJS web app.

The guides on the AngularJS website are good starting points but I find that a book can go into much more detail and provide more examples that fully explore each area of the AngularJS framework. Physical books also serve as a nice break from the screen and when written well, they let you mentally figure out what the code examples will do.

“[Mastering Web Application Development with AngularJS] definitely helps a beginner and serves as a handy reference for the experienced developer. Despite having done some work on AngularJS after reading a book and online courses, this book still provided me with so much information […] that I feel I should have picked up this material first.”

— Arun Mahendrakar, 5-star review of Mastering Web Application Development with AngularJS, amazon.com

Pro AngularJS

This book was recently published and has up-to-date code examples. It’s a heavy book and covers everything: directives, services, controllers. It has a nice dedicated section for defining AngularJS services and providers. It’s fantastic.

Here’s an excerpt from Chapter 9 – The Anatomy of an AngularJS App describing dependency injection:

The place to start is to understand the problem DI sets out to solve. Some of the components in an AngularJS application will depend on others. In Listing 9-4, my controller needs to use the $scope component, which allows it to pass data to the view. This is an example of a dependency — my controller depends on the $scope component to perform its work.

Dependency injection simplifies the process of dealing with dependencies — known as resolving a dependency — between components. Without DI, I would have to locate $scope myself somehow, probably using global variables. It would work, but it wouldn’t be as simple as the AngularJS technique.

A component in an AngularJS application declares its dependencies by defining arguments on its factory function whose names match the components it depends on.

As you can see, the explanation is quite clear, perfect for anyone new to AngularJS.

Mastering Web Application Development with AngularJS

This book gives a great overview and walkthrough of major concepts. One thing that’s impressive is that the book has unit tests for every example. Write out the code, write the unit test and run it and you can see exactly what’s going to happen.

What I really like about this book, aside from the extensive unit tests/examples, are the easy to understand explanations of core AngularJS concepts. For example, the $digest cycle and when you need to use the $apply method on scopes. If you understand them, you can improve the performance of your web app and you can avoid common mistakes.

Here’s an excerpt from Chapter 11 – Writing Robust AngularJS Web Applications, page 294:

The idea behind AngularJS model-changes tracking mechanism is based on the observation that at the end of the day, there is a finte (and rather small) number of situations where models can change. Those include:

• DOM events (user changing value of an input field, clicking on a button to invoke a JavaScript function and so on)
• XHR responses firing callbacks
• Browser’s location changes
• Timers (setTimeout, setInterval) firing the callbacks

Indeed if none of the earlier events take place (user is not interacting with a page, no XHR calls have finished, no timers firing) there is no point in monitoring the model for changes. There is simply nothing going on for a given page, the model isn’t changing, so there is no reason to re-render the DOM.

AngularJS starts its model-observing machinery if and only if it is explicitly told to do so. To bring this sophisticated mechanism to life, we need to execute the $apply method on a scope object.

As you can see, the explanation is clear and the book is full of gems like this.

AngularJS (O’Reilly)

Here’s an example of a good book that has outdated code examples. I would recommend getting this as a reference. This book is a solid reference that explains the major concepts so that you can apply them on your own.

I hope that they release a 2nd edition of the book with updated examples because O’Reilly books are usually pretty good.

Other AngularJS Books:

• AngularJS in Action
• Angularjs Web Component Development: Building, Reusing And Exporting Ui Components With Angularjs
• Instant AngularJS Starter
• Recipes with Angular.js

AngularJS libraries and modules:

• ux-datagrid: a highly performant list and data grid, designed for mobile devices.
• restangular: service that makes creating proper REST APIs easy.
• angular-translate: add multiple languages and translation to your app.
• angularfire: use this library to interact with your Firebase backend database.

News:

• AngularJS support in PhpStorm: autocomplete, custom directives, refactoring, routing and URL template support are now in JetBrains’ wonderful IDE, PhpStorm.
• AngularJS 2.0: implementation of the next major version is beginning, mobile-first and using ES6 (ECMAScript 6). More modularity is planned and a less complex dependency injection system is being designed.

AngularJS Meetups:

• Boston, MA
• Mountain View, CA
• New York City, NY
• Toronto, Canada
• London, UK

Top 15 Awesome Tutorials to Learn AngularJS

AngularJS 1.x is more stable and mature and while Angular has taken off, there are many projects still using AngularJS. It still makes sense for web developers to learn AngularJS. In a few years that may change, but for now, AngularJS 1.x is still a viable alternative to React, Angular 7, Vue.js and Ember and other frontend web frameworks.

If you’re interested in other Angular and AngularJS resources, check out the Learning AngularJS page.

Here are the top 15 awesome tutorials to learn AngularJS:

1. The Official AngularJS Tutorial
   In this tutorial provided by the team that developed AngularJS, you will learn how to create templates, components, and how to work with two-way data binding. Dependency injection is also introduced in the tutorial.
2. Year of Moo: AngularJS Guide
   In this guide from Year of Moo (which I’ve linked to before for their unit testing guide for AngularJS), you will learn how to create components and services and how models and directives work. It goes in depth on how modules are created within AngularJS for organization of projects.

3. Beginning AngularJS in 7 days
   While the website has a lot of ads, the information on it is very good and is a good beginner tutorial for AngularJS. It goes into specifics on how to use REST resources with the $resource module and how to make HTTP requests with $http.
   
   What I like are the sections on filtering and sorting tables and form validation, two common tasks in web frontends.

4. Todd Motto’s AngularJS Guide
   Todd Motto is one of the top AngularJS writers and developers. His complete guide to AngularJS is also a good reference when working on a project. It begins with an overview of engineering concepts in JavaScript frameworks, moves on to services and controllers and goes in-depth on routes and the $http and $resource modules.

5. AngularJS: Getting Started (PluralSight)
   The courses on PluralSight are all very high quality and easy to learn from. They even offer an AngularJS quiz so you can gauge what your own skill level. In this course they show you how to build controllers and directives. In particular it shows how to use the ng-click, ng-repeat, and ng-model directives.

6. Building a Mobile App with AngularJS 1.6 and Ionic
   In the video course, they cover how to build a mobile web app that can be deployed for iOS and Android using the Ionic framework.

7. Creating a Quiz with AngularJS
   Test yourself by building a quiz web app with AngularJS! Or just build the quiz app and learn how AngularJS works. What’s nice about this course is that it not only shows you how to build the components and controllers for the app, it also shows you how to style the app and how to move from one screen to another and to carry over data.

8. Build a GMail clone with AngularJS
   This is always fun, instead of a contrived project to build in order to learn AngularJS, you build a clone of an existing product. This was one of the best ways to teach game programming, making clones of popular games like Tetris or Super Mario World or Final Fantasy.
   
   In the web development world, making a clone of Instagram (as I’ve done for Elm) or GMail with AngularJS as this course shows you, is an excellent way to learn a frontend web framework.

9. Creating an app with AngularJS and Rails 4
   In this tutorial, you learn how to create a Rails API that can be used with AngularJS. It shows you how to use the $http HTTP module to make requests to a backend server.

10. Hands-on AngularJS Tutorial
    This tutorial shows you how to build an online movie booking app with AngularJS. There is a really good section on creating services.

11. Animate your site with AngularJS
    Animations are important for a smooth user experience, and this tutorial explains how CSS3 animations can be used to enhance your AngularJS app.
    
    Specifically, it shows you which directives have built-in animation support. Then it ends with walking you through how to create custom animations with JavaScript.

12. How to create animations with ngAnimate
    This article embeds code snippets of AngularJS so you can see the code in action and modify it and learn from it. It shows you how to animate a list of items, and how to show and hide validation messages for form validation.

13. AngularJS Routing Using UI-Router
    Routing in an SPA (single page application) is how a URL is translated into the right controller and view to display in the app. For example, a router determines how to route /hello to the HelloController and /world to the WorldController.
    
    The UI Router is more advanced than the router included with AngularJS which makes it powerful enough to be the default for many AngularJS projects.

14. Learn AngularJS 1.x Interactively
    CodeAcademy has an interactive way to learn AngularJS where you are given the code for the tutorials, it is explained and you’re free to edit it. The course is nice because it has both free form projects and quizzes to test your knowledge of AngularJS.

15. AngularJS Data Modeling
    This is an intermediate video course that shows you how to use caching in your models, how to add validation and manage validation through sub-classes.

Angular 5 vs Angular 4 vs Angular 2 vs 1: The complete list

In this article we compare:

• Angular 5 vs 4
• Angular 4 vs 2
• Angular 2 vs 1 (Angular vs AngularJS)

The newest version of Angular 5 is similar but also very different from the previous version of the framework, AngularJS 1.x. This reference page helps you learn the complete list of differences in the Angular javascript web framework. It includes some code examples of the differences and explanations of how they affect your web development with Angular.

IMPORTANT: At the moment, Angular 4 is the LTS version, LTS means Long-Term Support. The latest stable version of Angular is 5 which will continue to evolve. If you are building web apps or mobile apps using Angular, you will want to use Angular 4 because it will be stable in the long-term. You should be able to reliably build and support your Angular 4 app in the future; Angular 5 is more experimental.

Angular 2/4/5+ vs 1

Here’s a giant list of differences and we’re going to explore some of them:

Angular 2 works for more than just JavaScript: it works with TypeScript and Dart too. TypeScript is essentially JavaScript with static typing.

Angular 4 is geared towards TypeScript mainly; the focus of the documentation is on TypeScript rather than Dart or JavaScript. In Angular 4, TypeScript 2.1 and TypeScript 2.2 are supported which gives better type checking in VS Code.

Core modules are more modular, you can use 3rd party modules more easily (great example is the forms module which was separated from the core angular module)

AngularJS 2 has a new syntax for two-way data binding, uses the “banana in a box” syntax which looks like this:

• Component-based UI, directives in Angular 1 become components in Angular 2, and controllers simply disappear
• Events use a brand new event syntax, parenthesis around events in attributes
• Similar to React, properties are passed into components meaning that $scope is removed entirely from AngularJS. In Angular 2 v 1 you will create properties using the @Input decorator/annotation for a component (or directive)
• Dependency injection is more explicit with decorators/annotations, you can use @Injectable to declare a class that can be injected into other places, and you can use @Inject to declare that a function is dependent on another class
• Forms in Angular 4 are better structured modules and easier than ever with FormBuilder, FormGroup and Validators built-in
• Data binding makes implicit references to component variables meaning you can use {{hello}} instead of {{item.hello}}
• Filters have been renamed to Pipes
• Uses the let keyword to explicitly define template variables (just like JavaScript ES6’s let?)
• ng-app has been replaced with Bootstrapping, you explicitly call the bootstrapping method and pass in the module that is the application’s root module, also known as a main entry point
• CSS classes can be set based on variable bindings with a convenient syntax,
• ng-hide and ng-show directives no longer exist, you can use the “hidden” property binding to show or hide elements
• ng-href has been replaced by variable binding
• Router linking is done with the routerLink property
• The syntax for using ng-if has changed: *ngIf is the new way to use it
• ng-repeat has been replaced with *ngFor
• ng-src has been replaced with binding to the src property
• ng-style can be used with a convenient syntax
• The syntax for ng-switch has changed
• The filter and orderBy filters have disappeared for performance reasons; do any filtering and sorting of data within your components instead
• The number pipe/filter allows you to set decimal places now and you can use the percent pipe to display percentages data
• You can use ES6/ES2015 modules!! No need to wrap your modules in IIFE (immediately invoked function expressions)
• Defining a module’s imports/dependencies and what it exports/provides is far more explicit and less confusing
• CSS Stylesheets can be linked for specific components using the styleUrls property when defining a component, this makes sure styles aren’t leaking into other components and templates
• Zones and zones.js are used to keep track of execution context across multiple asynchronous events, making it far easier to reason about the internals of Angular 2 work in contrast to Angular 1 with its scope, digest and watchers
• Unit tests are easier to write in Angular 2 vs 1 because the rendering layer has been separated and they are faster
• Mobile web apps are supported with platforms and learn to create mobile/native apps using NativeScript
• Bower is no longer needed, everything is installed through NPM (Node Package Manager), or yarn if you’re more adventurous
• The HTTP module uses Rx.js for a Reactive data flow, this replaces promises and is great for realtime data retrieval
• Reactive programming is used everywhere; it’s used for events, HTTP requests, user input, and more. Data streams replace the $broadcast and $emit and you can create as many data streams as you need
• Angular 4 has a new animations package for creating smooth animations that give life and motion to your user interface.
• Below you will find more details about some of these items in the list of Angular 2 and Angular 4 vs AngularJS 1 differences.

Angular 2, Angular 4 , Angular 5 work for more than just JavaScript
Angular 2 works with three languages:

• TypeScript
• Dart
• Vanilla JavaScript (or ES2015/ES6 compiled with webpack to ES5)

The preferred language, at least judging by the API documentation and tutorials, is TypeScript. TypeScript is a language created by Microsoft that includes type annotations and other features, the language compiles down to JavaScript so that it is compatible with all web browsers. Angular 2+ uses annotations (also known as decorators) in TypeScript to make it simple to define components and other commonly used Angular classes such as NgModule and Directive.

There is an excellent article at Infoq on TypeScript and AngularJS 2.

Special Note: JavaScript ES6 is not the preferred language of Angular 2/4/5+, Reference API Docs are targeted towards TypeScript
You will want to use TypeScript instead of JavaScript/ES6 because the Angular team has decided that the core of the language will be written in TypeScript. This means that the best documentation is for TypeScript and that the tutorials and guides will mainly be written for TypeScript. This is not only true for the official/core Angular modules and reference documentation, it’s also true for community blogs and tutorials. Most of the community is using TypeScript.

Special Note #2: Bower Frontend Dependency Manager Has Been Deprecated
Instead of using Bower to manage frontend dependencies for Angular, the best practice now is to use NPM, the Node Package Manager to install Angular modules and libraries. Bower is used for older AngularJS projects but Bower is not used for Angular 2/4/5+ projects.

• Component-based UI, directives in Angular 1 become components in Angular 2, and controllers simply disappear
  AngularJS 1
  In Angular 1 you would use directives and controllers. They worked but weren’t as elegant as they could be. In Angular 2 vs 1 (and in Angular 4) they decided that controllers are redundant and you can use plain old JavaScript classes instead. This removes the need for $scope and simplifies your unit testing code because when you’re testing a controller, you’re just testing a simple class.

In Angular 2 vs 1 (and Angular 5), directives still exist but there is a specialized version of them called the component. In the majority of your Angular 2 coding you will be using components rather than directives. Components are a sub-class of directives so when you feel the need to use something more powerful and have a special use case, you can use directives instead. For example, the ngFor and ngIf components are actually directives because they have to hook into the life-cycle of a directive.

• Events use a brand new event syntax in Angular 2 vs 1, learn it
  This was tricky to wrap my head around at first. In Angular 2, when you are binding an event on an HTML element or component, you can use parentheses around the attribute:

Instead of saying “ng-blur”, you can skip that and use “(blur)”.

• Defining a module’s imports/dependencies and what it exports/provides is far more explicit and less confusing
  In Angular 2 vs 1, the module class ngModule is explicit in defining what it depends on and imports and what it declares for exporting. When you create an ngModule, you will typically be declaring components for export. This allows them to be used in other apps and modules. If you’re writing a module that has a class for API HTTP interaction, your module will depend on the HttpModule (here’s a way to unit test those modules). If you need the angular router, your module will be depending on/importing the RouterModule.

Here’s a more fully-featured example from the Angular 2/4+ API documentation showing what a module declares:

import { NgModule } from '@angular/core';
import { CommonModule } from '@angular/common';
import { FormsModule } from '@angular/forms';
import { AwesomePipe } from './awesome.pipe';
import { ContactComponent } from './contact.component';
import { ContactService } from './contact.service';
import { HighlightDirective } from './highlight.directive';

@NgModule({
imports: [ CommonModule, FormsModule ],
declarations: [ ContactComponent, HighlightDirective, AwesomePipe ],
exports: [ ContactComponent ],
providers: [ ContactService ]
})
export class ContactModule { }

• Mobile web apps with platforms and NativeScript
  You can create native mobile apps using NativeScript. It interfaces with the Angular framework to produce a native app that is as fast as other native apps. The way it does is is through the power of platforms.
• What are platforms? What is the platform browser?
  • A new feature in Angular 2/4/5+, platforms are interfaces to display and interact with the Angular code. The platform browser is how Angular code is displayed and interacted within the context of a web browser. There are two platform browser options; dynamic platform browser, and platform browser. Dynamic platform browser will compile your Angular code within the browser and then run the web app. The other option, browser platform, relies on you to compile your Angular code which the browser platform will then load and run, it skips the in-browser compilation step entirely resulting in better performance.
  • The mobile platform lets you display and interact with the Angular app through a mobile native app. The mobile platform is provided by NativeScript.

Deconstructing AngularJS Injector

Here’s a deconstruction of injector.js which contains all the functions and classes for dependency injection.

The Perils of Long Explanations and Examples in Comments
Before deconstructing the code, I had to figure out where the actual code was. The reason for this is that AngularJS’s developers tried to be clever and create their own custom documentation system. It looks kinda like JavaDocs, and it will extract the comments into a separate API docs file. The problem is that there is far fewer code in injector.js than comments, and the comments are longer explanations that are more fit for a tutorial or a manual than for a quick API reference, in other words, they belong in the docs/ folder as a text file.

The other problem with these comments is that they’re far away from their corresponding code. For example, there is a comment on the “get” method of $injector from lines 130-140 while the code is at lines 710-725, and line 786 (“get” is an alias for the “getService” method).

The original injector.js is 794 lines of code. When the comments outside of functions are removed, we’re down to 254 lines of code, you can see the result here: https://gist.github.com/omouse/7646610

Contracts or Static Types Would Help
The biggest problem I have with dynamically typed languages is that there are no type hints written out. With AngularJS’s implementation of dependency injection, the dependencies are passed around as the first items in a list followed by the function which dependencies on them. Instead of talking about them as functions (which have no dependency) or arrays where the last item is a function or calling it “array notation”, it would be helpful to create a type that represents them. Not a class, but a type. This would give it a name and make it easier to refer to.

Alternatively, contracts could be written for functions and methods that would state how to make a valid call to a function.

Being able to state function(x,y) requires that x is an Object that has the method “helloWorld” would be fantastic. It would save a lot of time reading through code. I know that not everyone does this, but Angular, at least in injector.js, is fond of creating objects that are plain Objects that don’t inherit or extend from other classes.

I guess this is why unit tests exist, but I find stating the contract right where the code is to be far more useful than referring back and forth between the code and the unit tests.

I’ll give an example here of what I mean:

function hello(name: String): void {
console.log(‘Hello ‘ + name + ‘!’);
}

function factory(name: String, factoryFn: Function): FactoryObject {
// …
}

function provider(name: String, provider_: FactoryObject or DIFunction): FactoryObject or DIFunction {
// provider_ is an object or a function that
/ uses dependency injection notation
}
Where the type FactoryObject is defined as an object that contains the $get method, and the DIFunction type is just a function or a list where the last item is a function. It would be much clearer to me to see what’s expected when using a function.

Anyway, let’s get to the deconstruction of injector.js…

Deconstruction: injector.js
Function: $injectorMinErr
Line 45 defines a new function using minErr called “$injectorMinErr”, the function returns an Error. The “$injector” argument is used as a prefix when errors are thrown. At line 593, in “createInjector”, the “$injectorMinErr” function is called, the arguments are: the type of error, the format string, and anything after that are variables to use in the format string. The format string uses {0}, {1}, … {N} for substituting variables. You can view the docs and code for minErr here.

Function: annotate
Lines 41-44 are used by the “annotate” function (line 46). They’re regexps, regular expressions.

FN_ARGS searches for a multi-line string that matches the text “function” and has a match group for function arguments. For example, it would match “function hello(first_name, last_name) { … }”, and the match would be “first_name, last_name”.

FN_ARG_SPLIT is used to split the arguments list into separate arguments, as an example, “first_name, last_name” would be split into the array [“first_name”, “last_name”].

FN_ARG has two match groups, one for the underscore and the other for the rest of the name of the argument. For example, for “param” the match groups would be “” and “param”.

Line 46-74 is the “annotate” function. The only argument it accepts is “fn”, a function or an array. It returns a list of dependencies, stored in the local variable $inject.

If “fn” is a function, we check to see if it has dependencies stored in “fn.$inject” as a list, at line 53. If the dependencies aren’t already stored, some cleverness happens. When “toString” is called on a function, the code for the function is returned and the code itself is anazlyed. The function name and arguments are extracted straight from the string representation of the function, using the previously defined FN_ARGS, FN_ARG_SPLIT, FN_ARG and STRIP_COMMENTS (lines 56-59). The arguments have any leading underscore removed (using FN_ARG). All function arguments are pushed into the dependency list, $inject.

Lines 66-72, if an array or other object is passed in, it uses “assertArgFn” to ensure that “fn” uses the array notation for a function, where last item in the array is a function, and the arguments up until the last argument are dependencies that need to be injected. If “fn” is some other object, an error is thrown by “assertArgFn”. If the array is valid, the items 1 to N-1 are stored in $inject, at line 69.

Function: createInjector
Line 576 defines the createInjector function. It accepts one argument, “modulesToLoad” which is an array of module names to load. It returns “instanceInjector” which is an object returned from the “createInternalInjector” function.

Line 580, the modules that have been loaded are stored in “loadedModules”. These are

The createInjector function has quite a few internally defined functions, the power of closures and first-class functions at work.

These internal functions are:

• supportObject(delegate)
• provider(name, provider_)
• factory(name, factoryFn)
• service(name, constructor)
• value(name, val)
• constant(name, value)
• decorator(serviceName, decorFn)
• loadModules(modulesToLoad)
• createInternalInjector(cache, factory)

Lines 581-590, the “providerCache” is created with a single property, $provide which contains references to various internal functions.

In line 595, the “instanceCache” is created.

Lines 591-594, the “providerInjector” is created and set as the providerCache’s $injector property. The factory method passed to “createInternalInjector” throws an error if the provider doesn’t already exist in the cache.

Contrast this with lines 596-600, the “instanceInjector” is created and set as the instanceCache’s $injector property. The factory method passed to “createInternalInjector” retrieves an existing provider (using the providerInjector’s get method) and then returns an instance created using the provider’s $get method (line 599).

Line 603, we’re looping through each module in “modulesToLoad” and using the the instanceInjector to invoke each module function.

We’ll go through the internal functions in order.

createInjector.supportObject

Line 611, the “supportObject” internal function accepts a function as an argument, “delegate”. The “supportObject” function is currying, it returns a function which accepts two arguments, key and value. When key is an object (see isObject), it loops through each property key/value (using forEach and reverseParams) of the object and calls the “delegate” function with the property key and value.

Otherwise, the key isn’t an Object and we pass the key and value argument to the “delegate” function.

Example:

function myDelegate(key, value) {
console.log('myDelegate, ' + key + ': ' + value);
}
var mySupportFunc = supportObject(myDelegate);

mySupportFunc('some key', 'some value'); // this will call myDelegate('some key', 'some value')
// the next call will loop over the keys/values and pass them to myDelegate
mySupportFunc({
'a': 1,
'b': 3,
'c': 5
});
// is the same as:
// myDelegate('a', 1);
// myDelegate('b', 3);
// myDelegate('c', 5);

createInjector.provider

Line 621, the “provider” internal function accepts two arguments, “name” and “provider_”. “Name” is a string, while “provider_” is either a function, an array, or a Provider object. When “provider_” is an array it’s actually a function with dependencies and uses dependency injection (lines 623-624). If it’s a Provider object, then it needs a “$get” factory method (lines 626-627).

Line 622 ensures that the “name” doesn’t have a “service” property set.

The “provider” function adds the “provider_” to the provider cache and returns the now-cached provider, line 629.

createInjector.factory

Line 632, the “factory” internal function accepts a “name” and a “factoryFn”. “Name” is a string, while “factoryFn” is a factory function. The function calls “provider” and passes both arguments to it (the factory function is used as the “$get” property of a new object).

createInjector.service

Line 634, the “service” internal function accepts a “name” and “constructor”. It returns the the result of calling the “factory” internal function, passing “name” and a factory function (a function with dependencies) to it. The factory function depends on the $injector and passes the “constructor” function to $injector.instantiate (defined on line 770, part of the object returned from createInternalInjector).

createInjector.value

Line 640, the “value” internal function returns the result of calling the “factory” internal function, passing in the “name” (a string) and the “val”. The “val” argument is wrapped in “valueFn” which returns a function that returns the given value, see line 383 in Angular.js.

Here’s an example of how to use valueFn:

// valueFn(value)
var getNumber = valueFn(3);
console.log(getNumber()); // prints 3 on the console

var getName = valueFn('Some name');
console.log(getName());

// getName could have been defined as:
// function getName() {
// return 'Some name';
// }

and here’s how the “value” internal function works:

var myValueFactory = value('hello', 'world');
// Evaluation/trace of the call to "value":
// value('hello', 'world')
// => factory('hello', valueFn('world'))
// => factory('hello', function() { return 'world'; })
// => provider('hello', {
// $get: function() { return 'world'; }
// })
// => { $get: function() { return 'world'; }
console.log(myValueFactory.$get()); // prints 'world'

createInjector.constant

Line 642, the “constant” internal function takes two arguments, “name” and “value”. It stores the “value” in the $injector’s providerCache and the instanceCache. The “name” argument should be a string, while the “value” can be any type of object.

createInjector.decorator

Line 648, the “decorator” internal function takes two arguments, “serviceName” and “decorFn”. The “serviceName” is a string, it’s joined/concatenated with the providerSuffix so that the provider can be retrieved from the providerInjector (line 649). The $get method of the provider is stored (line 650), so that it can be decorated (similar to how Python decorators work) in lines 652-654. The instanceInjector and its “invoke” method is used (defined later on, line 727). The decorated $get function returns the result of invoking the decorator function, “decorFn” with the original $get method passed in as a $delegate to “invoke”.

createInjector.loadModules

Line 661-702 defines the “loadModules” internal function. It accepts one argument, an array of modules to load. It loops through each item in the array, and if it’s a string it uses angularModule to find the existing module function to load. It also finds all of the modules that the module requires (stored in the “requires” property), lines 668-677. If the item is a function or array (a function with dependencies), there’s no need to search for an existing module, it’s used as is.

The checks for each item are wrapped in a try-catch block which throws an error (using the $injectorMinErr function) that displays the stack trace, lines 685-698.

createInjector.createInternalInjector

Finally we get to the good stuff, line 708 to the end of the file, the function that actually creates an injector (at least internally). The function accepts a cache object and a factory function.

It contains the following inner functions:

• getService(serviceName)
• invoke(fn, self, locals)
• instantiate(Type, locals)

createInternalInjector.getService

Lines 710-725, the “getService” function returns a service from the cache passed to createInternalInjector. If the service doesn’t exist yet, the factory function passed to createInternalInjector is used to create the service and the cache is updated.

Learning AngularJS Testing video course

In 2015, I published the course “Learning AngularJS Testing” for Packt Publishing. In 2016, it was available on O’Reilly, Udemy (over 100 students) and Lynda.com (over 8000 views).

Table of Contents

• Setting up AngularJS for Unit Testing and End2EndTesting
  • The Course Overview
  • Setting up Karma and Jasmine with Grunt Task Runner
  • Setting up the Project Structure
  • Configuring Karma for Testing
• Testing Controllers
  • Testing Controllers That Have No Dependencies
  • Testing Controllers That Have a Dependency
  • End2End Testing Two Controllers
• Testing Directives
  • Creating an Accordion Directive
  • Unit Testing the Accordion Directive
  • An End-to-End Test of the Controller That Uses the Accordion Directive
• Testing $resource-based REST API Services
  • Creating and Testing the $resource Service Using Mocks and Spies
  • Creating a REST API NodeJS Server with Fake Data
  • Displaying Information with a Directive (from the Service)
  • Displaying Information with a Controller

Course Description

Learning Angular.JS Testing will help you to understand the different aspects involved in testing components, such as controllers and directives, with the help of a practical pizza ordering application right from the start, all the way to the end!

We’ll start by constructing a project structure of our app, before installing and initializing various helpful tools such as Karma, Jasmine, and Grunt. We will create controllers for ordering the pizza, reviewing the order and test them be it with or without any dependencies. Not only that, but we will also go through testing the navigation between two controllers with the help of End2End testing. After that, we will create accordion directives for our application, unit test them, and resolve the errors if these tests fail. Furthermore, we will modify our services to use the $resource module and help them interact with REST API web services, as well as unit test them with the help of mocks and spies. Finally, we will create and test directives and controllers which fetch information from the service and display it.

At the end of the course, you’ll be familiar with the various approaches involved in testing AngularJS code and will be able to implement them to develop bug-free and maintainable applications.

In SCRUM user stories are added to a product backlog, which are organized and prioritized by the product owner into a sprint backlog. The work that needs to be done is in that sprint backlog and is worked through and checked on daily through the daily scrum (usually conducted by a scrum master).

In traditional project management, the project manager would be responsible for planning the work to be done and would meet with stakeholders, they would also ensure the work is moving along. In SCRUM Agile, engineering leaders work with their team and product owners to plan the work including estimating the effort and timelines. During daily scrums, also known as daily standups, tech leads or other members of engineering leadership can lead the meeting.

Scrum Agile Methodology and Agile Software Development

1. User stories (and tasks, features, etc.) are written and added to the product backlog. These can be written by the team, the project manager, project owner, or any other stakeholder.
2. The Product Owner (or project manager) organizes the product backlog.
3. During a sprint planning meeting, user stories are selected for design and development and implementation and prioritized into the sprint backlog by the product owner.
4. The team works on the items in the sprint backlog with the Scrum Master helping to unblock team members.
5. Every day the team meets for a daily scrum meeting where they discuss what they are working on and if they are blocked or stuck on anything. This meeting is also known as a daily standup.
6. At the end of a sprint, there is a retrospective to see what went well and what needs to change.
7. Then the cycle begins again with new user stories added, existing user stories selected for development and so on.

What is the Product Backlog?

The product backlog is the list of potential user stories that can be worked on. The list can contain tasks bugs and features as well.

It can be organized by priority, by estimated dollar value, by urgency, by number of customers who would be happy, or by some other criteria.

What is a User Story?

In contrast to traditional project management where work packages are worked on, in agile you have a user story.

Each user story includes the user persona or actor. It also includes what they would like to be able to do in the system.

Examples of user stories

• As an author, I would like to write a book (for a word processor)
• As the system administrator, I would like to view all performance metrics of our active systems
• As an educator, I want to create a course outline
• As an teacher, when creating a course outline, I want to include PDF presentation files and images

Estimating Time/Effort to Implement a User Story

For each user story, it is important to understand the effort involved in implementing it. The team members who will be working on the user story should take some time to break it down into sub-tasks and estimate how much time/effort they will take. If a user story looks as if it will take more than one sprint to complete, it’s important to know this sooner and estimation is how we determine that.

Engineering leaders will review and adjust team members time/effort estimates to ensure that all parts of the work are considered in a user story. For example, ensuring there are unit tests, manual testing, and documentation.

Story points, t-shirt sizes or days or weeks?

Some SCRUM implementations use “story points” to estimate the effort, others use t-shirt sizing such as small, medium or large effort. Others will use the number of days it will take and include a buffer of days to account for unknowns such as QA or testing or debugging time.

In project management outside of SCRUM, the estimation will usually be in number of days as part of a Gantt chart. Story points and t-shirt sizing are less precise and this is on purpose. It helps the team understand that it is a rough estimate, an approximation of effort that could change as the tasks are implemented and the work progresses.

Estimate and re-estimate

For example: it is possible to initially estimate a user story as being 3 in terms of story points. After some investigation and some work, the user story is re-estimated as a 7 in terms of effort. At that point, it may be prudent to move the user story into the next sprint or to re-evaluate what other tasks are being worked on and shift resources.

What is a Sprint?

A sprint is a block of time during which user stories and tasks are worked on and (hopefully) completed. The user stories are selected from the product backlog and put into the sprint.

How long is a Sprint?

Sprints are typically 2 weeks long. Most companies and startups have settled on 2-week sprints and plan accordingly. However, various SCRUM books suggest that sprints can be between 2 and 6 weeks. The longer the sprint, the more planning goes into it.

It can take up to a few hours to plan a 2 week sprint, anything longer than that and you may want to adjust the length of the sprint or select fewer user stories to work on within the sprint.

Varieties of SCRUM Meetings

There are two major SCRUM meetings:

• Sprint planning meeting
• Daily stand up meeting

These are the SCRUM meetings that every company that says they are agile uses to plan, execute and monitor the work.

There are other meetings too, such as the SCRUM Retrospective. This meeting is, unfortunately, not seen very often. Sometimes it is called a post-mortem, and sometimes they are conducted only after disaster strikes. Sprint retrospectives can be seen as “closing a project phase” in project management.

If you have Sprint Retrospectives, ensure that the notes taken during that meeting result in something actionable. Too often, it’s seen as a way to close the sprint rather than as a way to improve the team and the work.

SCRUM Tools

The typical tools used for SCRUM Agile planning are:

• JIRA: the most popular package for managing projects, it enables a variety of ways to manage workflows.
• Pivotal Tracker
• Asana: project management tool that can be used with SCRUM agile workflows

There are some alternatives to SCRUM Agile:

• SEMAT
• Linear, uses “issues” rather than “user stories”
• Shape Up, tracks priorities and measures progress in six week cycles
• Kanban, based on visualizing work in progress on a board,
• XP (Extreme Programming)
• SAFe (Scaled Agile Framework)

Process design is something that not enough people engage in, because of the day-to-day work that needs to happen to make the current process continue to function.

Sometimes, we get tunnel vision and need a different getting things done process design, or perhaps we need to make the current process even more efficient. Whatever the case, designing better processes requires reflecting on existing business processes and coming up with improvements.

When you work on a project, you can break it down into steps. Let’s take the example of writing an article. You would break that down into three steps.

• Step number one is researching the information and facts and statistics for the article.
• Step number two is creating a first draft.
• And the third step is reviewing and refining that draft.

Process Design Example

Now let’s try a different approach to writing an article.

• We could do a little bit of research and then write a paragraph.
• Then we can do a bit more research and write the next paragraph.
• And then, we can go back and revise the first paragraph.

In the first process for writing an article we have the three steps of researching, writing and revising. Now in this second process for writing an article, we have an iterative loop of researching, writing and revising. Each step in the second process is each individual paragraph with the final step being the completion of the article.

Now, which process is more efficient? Which process is better designed? Well, it depends on what you’re trying to achieve.

Process Design to Get Things Done at Scale

If you’re trying to achieve scale, and have the resources to hire a team the first process design is better. You could have, let’s say, five researchers conducting research for many articles, and a whole newsroom of writers putting the facts together into many articles.

Then you can have a few editorial staff to help edit and revise and refine the articles before publishing. It may take a bit longer to write the article but the trade-off is worth it. You will have multiple articles ready.

Process Design to Get Things Done Quickly

If you’re trying to publish your article quickly, the second process is better. In this process you write a good first paragraph with great research and publish it. Then you can add more information and refine the article further. Most news sites and blogs do this in order to deliver breaking news fast.

I used writing an article as an example but designing a better process can apply to web development teams.

For instance, the idea of a full stack developer who can write both frontend and backend code has gained popularity. On top of that they’re expected to be web designers. However, this can be inefficient for larger teams, where specialization leads to more in-depth knowledge and possibly faster development.

Questions to Ask for Better Process Design

If you want to be a high performer in your field, you have to optimize your processes.

Think about your business and personal life. What processes are you using to get things done? Choose one and try to design a better, more effective and maybe more enjoyable process.

• Is there a way to make whatever process you’re optimizing more efficient or less stressful?
• Is there a way to split up that process so that it can be delegated to others so you can scale up?
• Is there a way to batch many small tasks that are part of the process so you can maximize your time?

Think about these questions and take a bit of time to examine and improve your processes to get more things done.

The Scenario: You’re the team lead

Let’s set up the scenario:

• You’re a lead on a web development team, and one of the junior developers has been working on some code.
•  It’s time to check their progress in more depth.
•  The code they wrote after a few days is okay but could use some improvements, such as adding more tests or matching the code style, or there could be a significant architectural improvement.
•  The junior developer is waiting for you to see how much rework they must do.

Are you a fixer? Or a coach?

If your first reaction is to jump in and fix their code, then you might be a fixer. A fixer is a manager or leader who wants to fix things to meet their high standards.

Jill Geisler discusses this in her article, “Are you a coach or a fixer?” and says, “Fixing is a temporary solution to an ongoing issue. Your employee or staff isn’t really learning when you fix. If anything, they are learning to rely on you to upgrade things to your satisfaction, while not growing their own skills.”

For example, in our scenario, the junior developer must remember to follow the naming conventions of the project. When you’re a fixer, you’ll fix their code to match the naming conventions. The junior developer will have learned nothing from this and will not develop the proper habits. They will come to rely on you to fix their naming convention problems in the future.

As a team leader, you must move away from the fixer mentality and into a coaching mentality. When coaching, you are developing someone’s skills and knowledge so that they can become more developed as a person and as a team member. On a software development team, you train your team to become better programmers. You are training your team to rely less on you by internalizing the right things to do.

Let’s return to our scenario.

You choose not to be a fixer; you step away from your keyboard and walk over to the junior developer. You get into the coaching mindset, sit beside the junior developer, and pull up the code on the screen. You walk through a few spots you think need work and ask them questions about the code.

Your goal is to lead them to find the better solutions for themselves and to figure out what improvements can be made. You lead them with a question such as, “have you considered an alternative approach?” Another good question is, “what was your thought process when writing this part of the code?”

And just like that, the junior developer has started gaining the skills needed to become more mature and capable.

A coaching mindset leads to skill acquisition. A fixer mindset leads to stress.

The best teams, whether sports teams or software development teams, have coaches. Coaching your team is far more effective than simply fixing their problems. One of the coaching bonuses is that you’ll feel less frustrated and less stressed. Team leaders can apply coaching skills daily and see their teams grow as developers and people.

This is similar to the conceptual shift to map/reduce thinking. But it takes those ideas further. You’re doing a map/reduce and then connecting the data to other filters and maps and creating multiple pipelines so that the right data gets to the right place.

For example, when a user clicks a button in a web app (or desktop app), that click event can be propagated through the system to all the subscribers that are listening and waiting for that event.

When working with the Angular frontend web framework, that latest versions are now using RxJS, the reactive programming library for JavaScript. It is the underlying framework for working with events. Previously in AngularJS 1.x you would use the broadcast and emit system to act on events.

Reactive Programming

With Reactive, you have a generalized library for working with observables and subscribers which can work well with other libraries and other JavaScript code.

In Ruby and Python, other concepts have held sway for a long time, so here are some examples on how to do reactive programming Ruby and how to do reactive programming in Python.

These are basic examples of filtering and creating observables, think of them as the equivalent of “Hello World”.

You can find a list of other languages that are supported with a reactive library on the ReactiveX website.

RxJS: JavaScript Example

Let’s begin by creating an observable:

var numbers = [1,2,3,4,5];
var all = Rx.Observable.from(numbers);

This observable is being created from a list of numbers. It’s possible to convert other types of lists into Observables.

Next we want to print out all of the numbers. We do this by subscribing to the observable with a function:

all.subscribe(function(number) {
  console.log("The number is " + number);
});

Right now this looks like a for loop, but remember that observables are meant for events in time. There could be milliseconds or seconds or minutes between each event that the observable is emitting to the subscriber.

Next, let’s create a new flow of data by filtering all of the numbers so we only receive even numbers in our subscriber:

var only_evens = all.filter(function(number) {
  return number % 2 === 0;
});

Again we print out the numbers and you will notice that we only print out even numbers:

only_evens.subscribe(function(number) {
  console.log("This number is even: " + number);
});

That was a simple example of creating two flows; one with all values, and a filtered flow of values.

Now imagine in your next web app, a user does any action. That action is sent through an observable and we can have a subscriber that waits for a specific button click action or subscribes to any button click.

Then imagine a 3rd party API or REST API whose response you’re waiting on, you are subscribing to the observable and filtering for certain responses straight for display in your user interface.

There is a lot of power in reactive programming and it makes diagramming the process of UI event flow much easier. Each new user interaction or display is just another flow of events to be filtered and subscribed to.

RxRuby: Ruby Example

Now let’s do the same thing but this time in Ruby with RxRuby.

First we create the observable and print out all the numbers:

require 'rx_ruby'
numbers = [1, 2, 3, 4, 5]
all = RxRuby::Observable.from_array numbers
all.subscribe do |number|
  puts "The number is #{number}"
end

Then we filter the observable and print out the filtered values:

only_evens = all.select(&:even?)
only_evens.subscribe do |number|
  puts "This number is even: #{number}"
end

One interesting note when creating your observable, you should use from_array because simply using from will only emit the flow of values once. I’m not sure why RxRuby has this particular behaviour.

RxPy: Python Example

And now let’s do it all over again but with Python and RxPy.

We create the observable with all values and print them out:

import rx
numbers = [1, 2, 3, 4, 5]
all = rx.Observable.from_(numbers)
all.subscribe(lambda number: print('The number is {}'.format(number)))

And again we filter the observable’s values to a subset and print those out as well:

only_evens = all.filter(lambda number: number % 2 == 0)
only_evens.subscribe(
    lambda number: print('This number is even: {}'.format(number)))

Please note that in both the JavaScript and Python examples we used the operator “filter”, but in the Ruby example we used “select”. If you try and use “select” in the Python example you will get a list of boolean values instead of a properly filtered the list. There are a few API differences like this between each library.

Conclusion

This was a short introduction into reactive programming. There is a lot more you can do and here I wanted to demonstrate that it is possible to use Rx with more than just JavaScript. I have seen it used in Java and JavaScript and thought it was time to write more examples of Ruby and Python so others can see that those languages can also benefit from Rx and Reactive programming.

I’ve seen four ways of dealing with slow-running tests in Rails:

1. Start deleting tests (or mark them as skippable): the drawback is that your test coverage drops, and you may end up with more bugs
2. Use more integration tests rather than unit tests: this exercises a lot of code paths so test coverage won’t drop too much, but in the end, most developers will write integration tests that exercise the “happy path”
3. Run your tests in parallel
4. Run only the tests that were most likely to be affected by recent code changes

The latter two methods are much better than the first two.

My favourite so far is #3 because it still runs all of the integration and unit tests that you have, but makes use of the fact that you can offload the work of running tests to multiple machines. Everyone now has access to multiple machines through AWS (Amazon Web Services), Microsoft Azure, or RedHat’s cloud. You can run parallel tests on your local development machine.

Run Your Ruby on Rails Tests In Parallel (Rails 3, 4 and 5)

• Parallel_tests is a Ruby gem for Ruby on Rails that lets you make multiple test runs in parallel.
• How to set up parallel tests in RSpec across multiple machines using Jenkins CI.

The idea is that each test file will be run in a separate process.

If you have 5 tests that have a duration 2 minutes, in sequential runs it will take 10 minutes to run the whole test suite. If you have 5 processes available for parallel runs, it will only take 2 minutes to run them.

That’s a huge difference, meaning you can conduct 30 test runs in an hour rather than 6.

Rails 6: Parallel Testing is Built-In

The latest version of the Ruby on Rails framework, Rails 6, has parallel testing built into its core. It uses threads or processes for parallel test runs. The way it works is that you re-open the ActiveSupport::TestCase and add just one method call to it:

class ActiveSupport::TestCase
  parallelize(workers: 2)
end

Additionally, you can override the number of parallel workers by providing the environment variable PARALLEL_WORKERS when you run “rails test” like this:

PARALLEL_WORKERS=5 rails test

Splitting RSpec test files into multiple files

However, some of your Rails tests may take longer; within the spec, you could have multiple test cases and contexts that are taking too long. For instance, one test context with a few tests could take 1 minute, while the rest of the test cases in that file only take 10 seconds. At that point, you can split the long-running test context into multiple files.

Here’s an example of how that might look:

# spec/my_controller_spec.rb
describe MyController do
  context 'Slow tests' do
    it 'runs slowly #1' do
      # ...
    end
    it 'runs slowly #2' do
      # ...
    end
  end

  context 'Faster part of the test suite' do
    it 'runs in 10 seconds or less' do
      # ...
    end
  end
end

And now here’s how we could split that controller RSpec test file into multiple files.

# spec/my_controller_spec.rb
describe MyController do
  context 'Faster part of the test suite' do
    it 'runs in 10 seconds or less' do
      # ...
    end
  end
end

# spec/my_controller_slow_1_spec.rb
describe :MyControllerSlow1 do
  def self.described_class
    MyController
  end

  include_context 'my controller helpers'

  it 'runs slowly #1' do
    # ...
  end
end

# spec/my_controller_slow_2_spec.rb
describe :MyControllerSlow2 do
  def self.described_class
    MyController
  end

  include_context 'my controller helpers'

  it 'runs slowly #2' do
    # ...
  end
end

Most importantly, this works because we can override the class method described_class. This method is used by the rspec-rails extensions that make it easier to test Rails classes with RSpec.

In conclusion, by using shared contexts and helpers and by splitting files, you can optimize your Rails tests even further.

One more way to speed up Ruby on Rails tests

Above are ways to speed up tests today without doing too much work. The approaches above can yield quick wins, resulting in faster test runs, a more efficient developer feedback cycle, and slightly happier developers.

However, there is one more way to speed up tests: setting higher quality standards. Treat your tests as you would your code, and dive deep into what the testing frameworks offer and what optimizations they can give you to improve performance.

For instance, imagine you are using Rspec and Factory_bot and are trying to speed up a test suite with four test cases. Before each test case is run, there is setup that occurs. In this setup function, we create a few data models stored in the database. None of the test cases modifies the data. The first potential optimization comes from Factory_bot; you can construct the data models in memory rather than storing them in the database. The second potential optimization is to refactor all four test cases into a single test case and utilize RSpec’s custom expectation messages.

Now, imagine you have hundreds or thousands of test suites. Saving a few seconds here and there is good, but applying these optimizations across the entire test suite is even better. Unfortunately, this can be a time-consuming and challenging process.

So, if it takes a lot of effort to fix test suites written in the past, and we’ve already used the tactics for quick wins, what’s left?

The answer is fixing the future. You can achieve this by setting higher standards for all newly written test suites and optimizing them as soon as they are created.

In code reviews, review the test code as thoroughly as you would review the rest of the code.

The best reason to use GraphQL is if you have multiple frontend clients that communicate with your backend, and they each have differing data needs.

For instance, if you have a mobile application and a website that communicate with the same backend API, they will need different data. The mobile app may want to make as few requests as possible on startup, and batching the requests into one request is possible with GraphQL.

GraphQL Schema Example

import {
  GraphQLObjectType,
  GraphQLNonNull,
  GraphQLSchema,
  GraphQLString,
  GraphQLInt,
  GraphQLList,
  GraphQLID,
  GraphQLBoolean
} from 'graphql/type';
import co from 'co';
import mongoose from 'mongoose';

import models from './models';

var makeGQLString = function(desc) {
  return {
    type: GraphQLString,
    description: desc
  };
};

var makeNonNullGQLString = function(desc) {
  return {
    type: new GraphQLNonNull(GraphQLString),
    description: desc
  };
};

var makeGQLBoolean = function(desc) {
  return {
    type: GraphQLBoolean,
    description: desc
  };
};

var makeGQLInt = function(desc) {
  return {
    type: GraphQLInt,
    description: desc
  };
};

var listingType = new GraphQLObjectType({
  name: 'Listing',
  description: 'An event listing',
  fields: function() {
    return {
      id: makeNonNullGQLString('The id of the listing.'),
      slug: makeNonNullGQLString('slug'),
      title: makeGQLString('The title of the listing.'),
      description: makeGQLString('description'),
      description_html: makeGQLString('description_html'),
      category_id: GraphQLID,
      category_key: makeGQLString('category_key'),
      hashtag: makeGQLString('hashtag'),
      location: makeGQLString('location'),
      website: makeGQLString('website'),
      show_count: makeGQLInt('show count'),
      show_avatars_of_bookers: makeGQLBoolean('show_avatars_of_bookers'),
      show_tickets_sold_count: makeGQLBoolean('show_tickets_sold_count'),
      hide_date: makeGQLBoolean('hide_date'),
      capacity: makeGQLString('Capacity of the listing'),
      state: makeNonNullGQLString('State of the listing')
    };
  }
});

var eventType = new GraphQLObjectType({
  name: 'Event',
  description: 'An event',
  fields: function() {
    return {
      id: makeNonNullGQLString('Event id'),
      start_stamp: makeNonNullGQLString('Start of the event, timestamp'),
      end_stamp: makeNonNullGQLString('End of the event, timestamp'),
      city_name: makeGQLString('Event that the city is in')
    };
  }
});

var rootQueryType =  new GraphQLObjectType({
  name: 'RootQueryType',
  fields: {
    listings: {
      type: new GraphQLList(listingType),
      resolve: function(parent, args, ast) {
        return models.Listing.find().limit(10);
      }
    },
    listing: {
      type: listingType,
      args: {
        id: {
          name: 'id',
          type: new GraphQLNonNull(GraphQLString)
        }
      },
      resolve: function(parent, args, ast) {
        return models.Listing.findOne({ _id: args.id });
      }
    },
    events: {
      type: new GraphQLList(eventType),
      args: {
        listing_id: {
          name: 'listing_id',
          type: new GraphQLNonNull(GraphQLString)
        }
      },
      resolve: function(parent, args, ast) {
        return models.Event.find({ listing_id: args.listing_id });
      }
    }
  }
});

var schema = new GraphQLSchema({
  query: rootQueryType
});

export var getProjection;
export default schema;

We declare a root query that contains specific fields that are resolved by particular functions. The type of each field can be a list, non-null or other types. The resolve function is what fetches data from the database (or whatever caching layer(s) you have).

Within the GraphQLObjectType, we can define a structured object with more fields and define the types of those fields. This is in contrast to most REST API,s which provide no schema.

It’s also possible, as you define the fields, to provide descriptions of them. These descriptions can be displayed in a GraphQL API explorer. Since they are built-in and supported as part of the GraphQL specification, they can be supported by multiple libraries. In contrast, most REST APIs do not place descriptions near the fields within the code; they provide the descriptions (if at all) within separate documentation.

When building this prototype to provide a GraphQL interface to a MongoDB database, it was straightforward to declare the schemas for the models and then include them as part of the interface.

In the presentation I used PEP20, the Zen of Python, document as the starting point. PEP are the Python Enhancement Proposals and the Zen of Python was one of the earlier of these proposals.

The ideas within Zen of Python that I further examined are:

• Explicit is better than implicit
• Errors should never pass silently
• If the implementation is hard to explain, it’s a bad idea
• If the implementation is easy to explain, it may be a good idea

Jupyter Notebooks

Jupyter notebooks embodied parts of the Zen of Python, in particular that “explicit is better than implicit”. Jupyter notebooks (formerly know as IPython) has risen in popularity as it is being used for data science. Data scientists all rely on the notebook style to express their thoughts and to show their work and code. You can use Jupyter notebooks at Google Colab and Microsoft Azure, and see rendered versions of them in GitHub code repositories.

Peter Norvig, director of research at Google, uses Python heavily and created some IPython/Jupyter notebooks. You can see a rendered version of one of his notebooks on Github. What’s cool here is the space for explanation, including charts and graphs and mathematical formulas. The notebook is close to an interactive textbook.

Also, Jupyter notebooks supports many languages today including Python, Haskell, Ruby, R and Scala so you aren’t restricted to just Python. This is an example of the Python philosophy reaching beyond that one programming language and into other languages and domains.

Sphinx Documentation and Docstrings in Python

Within Python, you have the ability to create documentation using docstrings which aides in readability. Docstrings give developers a space to be more explicit about the intent of the code and to explain it.

The next project that I examined which embodies parts of the Zen of Python was the Sphinx documentation project.

Sphinx is how most Python-based projects generate their documentation and it generates it by using the docstrings that are part of the code. The default configuration of Sphinx produces good looking documentation with an index and a basic search engine.

Sphinx embodies and enables the idea that beautiful is better than ugly, that explicit is better than implicit. The markup language itself is complex, but not complicated, and is simple if you stick to the basics. So there are many points at which Sphinx embodies the Python philosophy.

The Design of Python APIs

Next there are the way Python APIs for libraries are designed. Python libraries have great API designs. I feel that the Python philosophy and the Zen of Python can take some credit for this.

The APIs of many Python libraries are sensible, they are explicit, they are explained well, they are readable. The Python philosophy strongly influence how Python libraries are written.

And this is just in Python, let’s see where else the Python philosophy has made an appearance.

JavaScript and the Zen of Python

In JavaScript, the Python philosophy has had a weaker influence. Though that may start to change, since there are Python developers who are switching to JavaScript and we should see more well-designed libraries that value readability, beauty, simplicity and so on.

One example that I want to point out is Express.js. Express is a Node.js-based web framework. In their code they have a router which dispatches from a url pattern to a controller. You may know this as the urlconf file in Django or as a url resolver.

JavaScript even with the additional new language features of ES2015 is very un-Pythonic. However, it still can express some of the Python philosophy, though at a higher level. JavaScript is made for interactivity which makes it ideal for interactive explanations of code.

We can see this at Khan Academy which uses interactive lessons to teach students. We can see it with specific libraries like d3 used for data visualizations that are created by the New York Times and other newspapers. As mentioned, Jupyter allows charts to be made and in the browser they’re rendered with JavaScript. Also, Sphinx.js was made to improve Javascript’s documentation.

So the story in JavaScript is that it’s difficult to express the Python philosophy except at a very high-level in terms of documentation and user-friendly tooling and better API designs.

Ruby and the Zen of Python

However, Ruby is a different story. Through my career I have encountered Ruby code that bears a strong resemblance to Python.

For example, pulling function parameters into their own local variables:

def before
  hello 'world', 123
end

def after
  name = 'world'
  repetitions = 123
  hello(name, repetitions)
end

This greatly improves readability and makes it simple to debug whether you’re using a proper debugger or print statements. Since the variable parameters are separate from the function call, you can print their values out before the function is called.

Another example of explicitness and readability in Ruby code that I recently worked on was for testing method calls.

Originally we had many tests relying on our test library to ensure that the method calls were accurate and had particular arguments passed to them. It was tedious work because we had to find a specific method call in the list of method calls and then test the arguments.

To match a list of method calls, it looked something like this:

describe 'method call test' do
  it 'calls the method with 1, "abc", :hello' do
    expect(MyClass.method_calls).to eq []
    MyClass.do_things
    expect(MyClass.method_calls).to eq([
      [:abc, [1, 2, 3, '127.0.0.1', 'b1d4a2', '45be3f']],
      [:def, [3, 2, 3, '127.0.0.1', 'b1d4a2', '45be3f']],
      [:efg, [3, 2, 4, '127.0.0.1', 'b1d4a2', '45be3f']],
      [:hij, [3, 2, 4, '127.0.0.1', 'b1d4a2', '45be3f']],
    ])
  end
end

The new code now looks like this:

describe 'method call test' do
  it 'calls the method with 1, "abc", :hello' do
    expect_do_things_call(:abc,
      :id => 1, :other_id => 2, :thing_id => 3, :ip_address => '127.0.0.1',
      :role_id => 'b1d4a2', :category_id => '45be3f'
    )
    expect_do_things_call(:def,
      :id => 3, :other_id => 2, :thing_id => 3, :ip_address => '127.0.0.1',
      :role_id => 'b1d4a2', :category_id => '45be3f'
    )
    # ...more of the above...
  end
end

The upside is that the test is explicit in the arguments and parameters that will be passed to the method. The expectation is far more clear and it increases readability.

Tests are used to double-check our work but if the code in those tests is dense, nested and hard to understand, then we can’t double-check our work effectively. The Python philosophy of readability and explicitness is at play here.

I would also say that the “flat is better than nested” and the “sparse is better than dense” concepts in the Zen of Python are also at work.

Conclusion

While some of the ideas here are essential to good software development and software engineering, they are uniquely embedded within Python.

These ideas are embodied within the Zen of Python which sets the standard and affects Python and its community. With Sphinx and Jupyter, the Python philosophy becomes more widespread.

(2024/03/10): This piece of advice for ICs (Individual Contributors) still applies. Having a short list of topics can focus a 1-on-1 towards a particular tactical issue or for a strategic discussion. As an engineering manager, I appreciated it whenever my reports would bring a topic to the 1-on-1, whether it was a technical topic or a career or mentorship topic.

What is a Cause and Effect Diagram?

A cause and effect diagram is drawn by starting with the effect that is being produced. This could be a bug or a problem or some other issue with the software or IT project.

Then the individual or, preferably, the team comes up with possible causes. This list of causes is investigated, one at a time, and sub-causes are determined.

If there are no sub-causes, then it’s likely that this wasn’t the actual cause of the effect. If there are sub-causes, and sub-sub-causes, then you’re on the right track and will find the actual cause of the effect.

Why are Cause & Effect Diagrams Also Known as Ishikawa Diagrams?

Kaoru Ishikawa created the concept of the cause and effect diagram in the book “Guide to Quality Control”. The book was first published in 1968 and republished in the 1970s and 1980s.

Dr. Ishikawa wrote a few books on quality control and Ishikawa/Cause & Effect diagrams are part of them.

The book “Guide to Quality Control” features more than the cause & effect diagram. The book also features control charts, histograms, scatter diagrams, check sheets and Pareto diagrams.

Usefulness of Cause and Effect Diagrams

Cause and effect diagrams are useful because they let you list out all the causes that you and/or your team can think of.

When investigating a problem, people will usually start with one or two causes and investigate those causes. Then they will think up more potential causes if those first few causes were not at the root of the effect and try to investigate those. In other words, they start with A and B, investigate those and then move on to investigate C and D. This approach works.

However, it is not as effective as the cause and effect diagram approach because it is not systematic and it does not outline common causes that should be investigated. The simplistic approach can mean wasted time and energy when the causes aren’t the root cause of the effect. In an IT environment or software development project, it can mean debugging for hours and getting no closer to a solution.

In contrast, by stepping back and thinking of categories of potential causes and walking down each potential cause as they are investigated, you can narrow down the root cause much sooner. As an additional benefit, you can share this with other stakeholders on the project to keep them informed of the status and progress of an investigation.

Peer Review of Causes

Having a team think of possible causes is similar to the concept of peer review in software development projects. Where one person thinks of a few causes, another person will think of different causes. This gives a broader range of causes to investigate. Some of the causes can be eliminated as other team members may have already investigated them and found them to be dead-ends.

Listing causes together as a team can be a great exercise in collaboration.

Creating a History of Causes and Effects

If cause and effect diagrams are stored in the organizational knowledge base, they can help in the future.

If an effect sounds similar to another one, such as two bugs in the same component, the causes could be similar as well. Being able to search through previous cause & effect diagrams when a problem comes up may make it faster to solve the problem.

Cause and Effect Diagrams in Markdown or Plain Text

If you are in the tech industry, it is more common to store plain text or Markdown files and text-based formats are easier to search through and can be kept in a version control system.

Common Causes in the Marketing, Services, and Manufacturing Industries

Marketing Common Causes

In the marketing industry it is common to start with a set of common causes (8 P’s):

• Product/Service
• Price
• Place
• Promotion
• People/personnel
• Process
• Physical Evidence
• Publicity

Manufacturing Common Causes

In the manufacturing industry, they start with these causes (5 M’s):

• Machine
• Method
• Material
• Man Power/Mind Power
• Measurement

Service Industry Common Causes

In the service industry it’s common to start with these causes (4 S’s):

• Surroundings
• Suppliers
• Systems
• Skills

Common Causes in the IT and Software Development Industries

In the IT and software development industries, a list of common causes has not been developed. So the following is my attempt at creating the list of common causes when investigating problems in the IT and software development industries.

The common causes are MVPS for IT and MCPS for software development. They focus on the key causes such as Vendors or Code that are highly likely to be the cause of issues. They are compact (there’s only 4 of them, each) and a quick way to get started with using cause and effect diagrams.

MVPS: IT Project Causes

On IT projects these are good starting points for causes (MVPS):

• Material (includes hardware and software)
• Vendors (such as hardware/software vendors)
• People (your team members, coworkers and other stakeholders)
• Systems (used to control and manage IT functions)

MCPS: Software Development Project Causes

For software development projects, this set of common causes is solid foundation to investigate (MCPS):

• Method (include software development life-cycle, QA process, agile methodology, SCRUM methodology, Waterfall)
• Code (the source code and related artifacts such as packages and libraries and frameworks)
• People (your team members, coworkers and other stakeholders)
• Systems (the systems used for development, releases, deployments, testing)

Case Studies

When I wear my software developer hat and work on IT projects, I sometimes have to search for the cause of a bug in the software and fix it.

This requires using a root cause analysis to first find all the issues contributing to causing the bug. Then fixing them all either by writing more code, writing more tests, or by changing the design of the software.

However, before I can determine the cause and fix it, I have to figure out what the possible causes are.

By using a cause and effect diagram, I can list out as many potential causes as I can think of. Then I can investigate each potential cause and eliminate them until I get to the real cause (or causes). It may sound tedious but it’s the only way to be sure that some causes were not prematurely eliminated.

We use the MVPS (Material, Vendors, People, Systems) and MCPS (Method, Code, People, Systems) common cause categories as starting points in these case studies.

Low Developer Velocity: Causes and Effects

The following is a cause and effect diagram where the problem is low developer velocity.

The software development team as a whole is taking longer to ship working software. By using MCPS (Method-Code-People-Systems) as the starting causes we can dive deeper into the sub-causes.

Do remember that the causes are possible causes. Some of them are not causing the problem. In other cases, the causes are contributing but can be ignored because they are low risk for the project.

Every software development project is different and each problem can have one or many causes.

Exploring and Drilling Down into Sub-Causes

Sub-Cause Exploration: Development Servers Are Down

Let’s say we are exploring the “Development servers are down” cause under the Systems cause. We could split that further into checking whether the problem is with the internet connection to the servers, the firewall rules, domain name issues, hardware problems, billing issues, and so on.

Before we investigate those potential causes of the development servers being down, we first check if they are in fact down. If the development servers are up, we can eliminate this from the list of causes of low developer velocity.

Sub-Cause Exploration: Too Many Meetings

“Too many meetings” can be a real issue for mid-to-large sized companies and for enterprises. Checking the work calendars of coworkers, team members, and other stakeholders can quickly determine whether “too many meetings” is the cause of low developer velocity.

If everything is going right, many of the potential causes will be eliminated and you will quickly narrow things down to a few causes. Investigate and document them thoroughly and you will be prepared for future issues that are similar.

Sub-Cause Exploration: Too Much Technical Debt

For a startup, “too much technical debt” could be the root cause of many software development problems. At first, the velocity of the team in a startup will be high but as the startup grows, the technical debt will catch up to the team and slow them down.

Eliminating technical debt should always be a priority, but when trying to improve velocity, a more nuanced approach is required. What are the potential causes of technical debt? Moving too fast, poor architecture decisions, outdated technology, little or no testing, and so on. Again, most of these potential causes will be eliminated as causes of low developer velocity.

Server Procurement Takes Too Long: Causes and Effects

It can take too much time to procure a server. This is a common I.T. issue for organizations that are not using cloud server infrastructure. In rare cases, this problem also exists in cloud-using organizations but it is very rare (and mostly a matter of giving the right authorization/permission to the I.T., devops and development teams).

In the following diagram we start with the MVPS (Material-Vendors-People-Systems) causes since this is an I.T. issue. The potential sub-causes will be different for different organizations but you should notice a few that exist in your organization if you are not using cloud-based server infrastructure.

Let’s take a look at some of the potential causes. Procurement is not automated will typically be the cause in non-cloud organizations. In places where procurement is automated, server procurement time drops dramatically. In one case I’ve seen a 3-week procurement time drop to 2 days.

Sub-Cause Exploration

In a development agency, server procurement may take too long because the vendor’s billing process is convoluted or because clients do not want to pay for the server yet or because many approvals are required for procurement. For example, procurement approval could require project manager, account manager, IT and client sign-off because it affects the work schedule and what and when the client is billed.

Drilling Down Into More Sub-Causes

Let’s say that “client is not willing to procure the server yet” is an extremely likely cause of our problem.

What are the causes of this? Let’s drill down and figure out some more sub-causes.

• The client may not have budget for the server until after a certain date or project milestone
• The client is unsure of the success of the project and whether it will be launched and may not need the server at all
• The client is not technical and does not see the need for a server yet
• The client already has their own servers
• The client does not want to use our private cloud infrastructure
• The client does not want to use public cloud infrastructure
• The client wants to build their own server infrastructure
• The client wants to determine all the hardware/software specifications of the server which takes time

We could go on and on and in a real-world scenario, your team can brainstorm more causes and work together on eliminating these potential causes as the source of slow server procurement.

Applying Cause and Effect Diagrams

As the IT and software development industries mature, you will see more and more quality control tools used. Cause and effect diagrams are the best first step to using other quality control tools since determining and fixing the causes of problem effects are something that developers, devops and IT teams do on a regular basis.

The MVPS (Material-Vendors-People-Systems) common causes for IT projects and the MCPS (Method-Code-People-Systems) common causes for software development will cover many of the causes of problems in the IT and software development industries.

MVPS and MCPS can be used to introduce the cause-and-effect diagram tool to the IT or software development team. Using a cause and effect diagram will have an immediate positive impact on productivity and increase organizational knowledge in order to prevent further issues.

Cause & Effect Diagramming Tools

You can create cause & effect diagrams (also known as fishbone diagrams) in any diagramming software that supports boxes and lines, you can create them in LibreOffice or PowerPoint or with online tools like Diagram.net and Miro.

Here’s how you can create a cause and effect diagram from a template using Miro:

And here’s how the template will look like in the Miro user interface:

From there you can change the text to match the concepts that you want to explore as the causes and effects for a particular situation.

You can create cause and effect diagrams using a text editor using plain text or Markdown:

Each “alpha” component of each layer has a checklist to see which progress level it is at. The levels range from 1 to 6 (or fewer depending on the alpha component).

For instance, the stakeholder alpha of the customer layer ranges from level 1 (recognized) to level 6 (satisfied in use).

In the project lifecycle, a software system will move through the various levels, starting from an architecture being selected (level 1) to a demonstrable prototype being developed (level 2), to being fully operational (level 5) and then eventually retired (level 6).

You can download the SEMAT kernel cards here to gauge your own project, feature or team.

Alpha States in SEMAT

There are three layers that the Alphas belong to and each one has a different set of progress levels.

Customer layer:

• Opportunity
• Stakeholder

Solution layer:

• Requirements
• Software System

Endeavor layer:

• Work
• Team
• Way of Working

I tried using the appropriate layers to check the health of a small task I was working on and after I was complete, used the SEMAT Kernel levels to see how the health/status had changed.

You Do Not Need to Use Everything in the SEMAT Kernel

Not all Alphas need to be applied. The small task was a new feature for exporting an Excel spreadsheet that I would work on and I had already discussed the specification with the client. Working on it on my own while tasking other team members with other tasks meant that I did not need the “Way of Working” or “Team” Alphas. Since this was only one feature, I would not need the “Software System” or “Stakeholder” Alphas either.

Therefore, the only Alphas/components I needed to check the health of the task were:

• Opportunity
• Requirements
• Work

Example of Using SEMAT

First Health Check of the New Feature

When I first checked, I got the following progress levels:

• Opportunity: viable 4/6
• Requirements: acceptable 4/6
• Work: started 3/6

To be at a certain progress level, I had to go through the short checklist on each card and find one that matched the current status of the project.

The checklist for “Opportunity: viable” is

• A solution has been outlined
• Indications are solution can be developed and deployed within constraints
• Risks are manageable

The checklist for “Requirements: acceptable” is:

• Requirements describe a solution acceptable to the stakeholders
• The rate of change to agreed requirements is low
• Value is clear

The checklist for “Work: started” is:

• Development work has started
• Work progress is monitored
• Work broken down into actionable items with clear definition of done
• Team members are accepting and progressing work items

(You can download the SEMAT kernel cards here to gauge your own project, feature or team)

The client brought up the idea of a new feature and we talked about what was required. I asked some questions to clarify what the client wanted. Those conversations were dealing with the opportunity & requirement Alphas. The client was satisfied with the spec for the new feature, which is why the opportunity was “viable” and the requirements were “acceptable”.

I had already begun to work on the feature, writing unit tests for related functions/classes and installing an additional library for creating Excel spreadsheets. There was a ticket in the ticket system for the feature which included the definition of done for the task. Therefore, I had “started” work.

Second Health Check of the New Feature

After some coding, I finished the feature and it was ready to be deployed to the staging server so that the client could take a look and offer feedback.

I took out the Alpha states cards and did another health check of the feature:

• Opportunity: viable 4/6
• Requirements: addressed 5/6
• Work: concluded 5/6

The checklist for “Opportunity: viable” remained the same as before.

The checklist for “Requirements: addressed” is:

• Enough requirements are implemented for the system to be acceptable
• Stakeholders agree the system is worth making operational

The checklist for “Work: concluded” is:

• Work to produce results have been finished
• Work results are being achieved
• The client has accepted the resulting software system

I implemented the feature according to the specs and at that point, the client needs to test it. This is why the requirements have only been “addressed” and have not yet been fulfilled; there may be bugs or the feature may need more functionality.

The work was “concluded”, since I implemented and tested the feature locally, fixed a few bugs and it worked. I am done working on the feature for now. However, I am waiting for the client to test the feature. That means the work cannot be at the final stage, “closed”.

After the client inspects the result, I expect the health of the feature to be at the final stage for “Opportunity”, “Requirements” and “Work”.

The Advantage of SEMAT

The advantage of SEMAT from this example is that it was flexible. Through this flexibility, I kept track of progress on the Alpha states that I needed. There was no obligation to use all of the Alpha states that the SEMAT software project management framework provides.

It is flexible enough that it was easy to see map the current status of tickets in the ticket system, as you can see in the second health check, the work was concluded; the ticket was marked as “needs testing”.

The flexibility lets you use it with Agile, SCRUM, and other methodologies.

JsonCpp is a good solid C++ library to work with. The code samples below show that you can easily work with C++ and JSON files. What’s interesting about JSONCPP is that Google uses it in their C++ library for their Data Layer API.

JsonCpp (also known as JSON C++) gives you the ability to open JSON configuration files. This means you can re-use configs and build a faster more performant project out of a frontend web development project. You can use C++ and Node.js and get the best of both worlds through sharing JSON files and JsonCpp!

JsonCpp is great to work with, however the API docs need more of a tutorial. There is a Repl.it instance with code samples that should serve as a better tutorial to help you use the JSON C++ library.

Saving an Object to JSON

Here’s a example of the JSON-specific code to save an object to JSON:

void AddressBook::JsonSave(const char* filename) {
    ofstream out(filename, ofstream::out);
    Json::Value book_json(Json::objectValue), contacts_json(Json::arrayValue);
    for (vector<Contact>::iterator it = contacts_.begin(); it != contacts_.end(); ++it) {
        contacts_json.append((*it).ToJson());
    }
    book_json["contacts"] = contacts_json;
    out << book_json;
    out.close();
}

Json::Value Contact::ToJson() const {
    Json::Value value(Json::objectValue);
    value["name"] = name_;
    value["phone_number"] = phone_number_;
    return value;
}

The above code is also a JSON C++ array example. We are using a vector for a dynamically-sized array, and we are using the JSON C++ arrayValue to extract a list from the JSON.

Loading JSON List/Vector of Objects

Here’s an example of C++ JSON for loading a list/vector of objects from JSON:

void AddressBook::JsonLoad(const char* filename) {
    ifstream in(filename);
    Json::Value book_json;
    in >> book_json;
    for (Json::Value::iterator it = book_json["contacts"].begin(); it != book_json["contacts"].end(); ++it) {
        AddPerson((*it)["name"].asString(), (*it)["phone_number"].asString());
    }
    in.close();
}

void AddressBook::AddPerson(string const &name, string const &phone_number) {
    Contact contact = Contact();
    contact.set_name(name);
    contact.set_phone_number(phone_number);
    contacts_.push_back(contact);
}

Explanation of Loading JSON C++ Code Example

Okay let’s break down the JSON C++ code above!

First we’re defining a method, JsonLoad, in the AddressBook class. It accepts a filename string argument:

void AddressBook::JsonLoad(const char* filename) {

Then we are creating an input stream using ifstream:

ifstream in(filename);

And we create a variable that will store the parsed JSON value:

Json::Value book_json;

Then all we have to do is pipe the input stream into the JSON value object. The JsonCpp library defines the >> operator and will take care of parsing of the input stream into JSON:

in >> book_json;

Interactive Code Example

Click here to try out and run the code with an interactive example on Repl.it.

The project was a software prototype for a restaurant ordering system. The goal was to design a user interface that allowed a customer in the restaurant to use their iPhone or some other smart phone to quickly order food.

For this prototype, I used Squeak Smalltalk, the Seaside web framework, and the Magritte meta-description framework.

Database in Smalltalk

As the database for this prototype, I used a class with only class methods. It used Dictionaries to store the data, just a plain key-store database.

I did not see a need to explore the object-oriented database systems available nor did I want to force the instructor of the course to setup an SQL database on his computer just to test this prototype.

Update 2019: Today I would create a database and connect the Smalltalk image and Seaside-based web app to a SQL or MongoDB instance.

Smalltalk Components and Tasks

There were several processes and screens that needed to be mapped to Seaside WAComponents and WATasks.

For example, placing an order is a process/task that will first force you to login, then allow you to browse a menu, and when you’re done ordering, it will place the order into the system for the kitchen staff to handle. Within that, there would be a “Login”, “Menu Selection”, “[Menu Selected] Items”, and “Review Order” screens/components. This mapping made it easy to figure out where to sub-class components and where to jump into different tasks.

Later on, I realized that Magritte might make things easier for me. I was trying to figure out how to model menu items with varying options, such as pizzas that came in only 2 different sizes, salads with a choice of dressings, and burgers with a choice of “toppings” (with tomatoes? with mayo?).

I wanted to write a generalized MenuItem class with a render method that could display check boxes or radio buttons depending on what was needed. This would have allowed me to add or remove option groups because what if the restaurant no longer wants to give people a choice of pizza sizes?

Then I realized that this was too customizable and cut back, because there was only one place where the menu item was viewed with its options.

In the end, I created a MenuItem class and then sub-classed it into PizzaItem, SaladItem, etc. with each one supporting different options.

In the ViewItem class, the renderContentOn: method would call other methods depending on the class of the menu item. So if the item’s class was PizzaItem, then renderPizzaItemOn: would be called.

Magritte’s Help

Magritte is a meta-description framework for the Seaside web framework. It lets you describe the model of the object you want to display and then use different views that render those descriptions. It makes it easier to display an object.

Instead of making an EditObjectView or a CreateObjectView, and then calling them, you just call the asComponent method on the Object, and Magritte will generate the HTML for you. The view that it uses is either a table- or CSS-based view.

I used Magritte to create the admin tool and the control panel where I modified the database. I defined the descriptions for all the objects and then let Magritte create the forms so I could create or edit them. For example, the Menu object had 2 class methods:

• descriptionName
• descriptionItems

Magritte will automatically look for class methods that start with “description” and use them as part of a default view (which is found in the class method description). On the control panel WAComponent, I rendered a list of menus and then had a link that said “Add Menu”, something like this:

renderContentOn: html
    html heading level1; with: 'Menus'.
    menus do: [ :menu | self renderMenu: menu on: html ].
    html paragraph: [ html anchor callback: [ self createNewMenu ]]

The createNewMenu class is where the Magritte-related magic happens and it looked like this:

createNewMenu
    | menu |
    menu := self call: (Menu new asComponent addValidatedForm).
    menu ifNotNil: [ menus add: menu ]

Simple to use as you can see.

Seaside’s and Smalltalk’s Awesomeness

Seaside web framework was wonderful to work with. I found it slow at first but now that I’ve grown more used to it, it isn’t that bad and I feel like I could write some simple web applications much more quickly in Seaside/Smalltalk than in Django/Python.

There is not too much to say, the awesomeness of Seaside is subtle, such as automatic session creation, composable components, proper tasks, etc.

Seaside’s Problems

Seaside’s documentation is lacking. I couldn’t find comments for some components that I thought I wanted to use.

Seaside also has a precedence problem when you are HTML rendering. I ran into this when I tried to use radio buttons. When rendering a view in HTML, the order of the methods called matters. The with: method should be the last call made to an HTML tag object since it renders the object. I had reversed the order and was creating radio buttons like this:

renderContentOn: html
    | group |
    group := html radioGroup.
    group radioButton
        with: 'Some Text';
        value: 'Hello'.

That’s incorrect because the radio button will be rendered before the value: method is called on it. What I really should have done was this:

renderContentOn: html
    | group |
    group := html radioGroup.
    group radioButton
        value: 'Hello';
        with: 'Some Text'.

Smalltalk is Multi-Platform

Overall, I found working with Smalltalk awesome. I was able to work on the project on multiple platforms and operating systems without any changes. Using Ruby or Python or other languages usually requires tweaking of package/module/library paths and settings, but with the Smalltalk images that wasn’t a problem.

On each platform all I had to do was install Squeak Smalltalk. Then I had to copy the image over to the computer and run it. No messy setup of databases, text editors, version control. Just download, install and start the Smalltalk image for the project!

Magritte helped me avoid writing redundant admin/database management code. So it saved time and effort to complete the project.

Update 2017: I currently work with Ruby on Rails, and previously worked on frontend web apps using React, Angular, Elm and Ember.js. Before that I worked with Django. After all these web app frameworks, I still remember the joy of using Smalltalk’s excellent IDE and the Seaside.st web framework.