Case study: analyzing the Walmart site performance

Walmart is one of the top USA e-commerce retailers. In 2016, they were the second after Amazon by sales.

In e-commerce, the conversion is directly affected by how fast the site loads. For many e-commerce companies, making the site faster by 1 second increased the conversion 1.05, 1.1, or even 1.2 times. That’s because the slower the site, the more users abandon it before it loads, and the lesser is the conversion.

Unfortunately, the Walmart site is pretty slow. In my tests, the content of the product page becomes visible only at the third second:

In comparison, for Amazon, the content gets visible at 1.4 seconds. The customer sees the product they came for twice faster!

Let’s analyze the Walmart’s site and see how we can improve the performance – and help Walmart earn more! I’ll use the Lumia 635 product page as an example.

Fix the invisible text#

The first issue with the page is that it gets more or less rendered at 2.3s, but the text isn’t visible until 3.0s:

This happens because Walmart uses a custom font, and by default, Chrome and Firefox won’t render the text until the font is loaded. This is how it looks live:

See how the page stays without the text for a second?

(Network throttled with the “Fast 3G” preset in Chrome DevTools)

Flash of unstyled text is when the text was initially rendered with a system font, but later gets re-rendered with a custom one (and jumps around the page). Wikipedia

Browsers delay rendering the text to prevent a flash of unstyled text (FOUT). However, this makes the content invisible for longer – and likely decreases the conversion!

To change this behavior, we can add the font-display: optional rule to the @font-face styles. font-display controls how the custom font is applied. In our case, it tells the browser to just use a fallback font if the custom one is not cached:

/* https://ll-us-i5.wal.co/.../BogleWeb.css */
@font-face {
  font-family: "BogleWeb";
  /* ... */
  font-display: optional;
}

Now, when a customer visits the page for the first time, they will see the text immediately, rendered in a fallback font. The browser will download the custom font in the background and use it for subsequent pages. The current page won’t get the custom font – this prevents the FOUT:

Now the text is visible immediately.

(Network throttled with the “Fast 3G” preset in Chrome DevTools. The CSS file was substituted with Fiddler)

Side note: single-page apps#

With font-display: optional, the font won’t be applied until the user reloads the page. Keep this in mind if you have a single-page app: navigating across routes there won’t make the font active.

Optimize JavaScript#

Another issue is that the page downloads around 2 MBs of gzipped JavaScript. That’s a lot:

JavaScript code is minified, so I’m only able to analyze it on the surface. Here’s what I found.

Use defer for the first bundle#

Most of <script> tags on the page have either the async or the defer attribute. This is good because the browser can render the page not waiting for these scripts to download:

The page has more scripts in different places, so that’s just an example

However, one large file – bundle.3p.min-[hash].js, 112.3 kB gzipped – doesn’t have either of these attributes. If it takes a while to download (e.g., the customer is on a bad connection), the page will stay blank until the script is fully loaded. Not cool!

To be honest, the bad connection could delay any non-deferred script, even the smallest one. So I’d try to defer as many scripts as I can

To solve this, add the defer attribute to this script tag too. As soon as all JavaScript that relies on bundle.3p.min-[hash].js is also deferred (which seems to be the case), the code will keep working fine.

Side note: performance marks#

In the screenshot above, there’s code that likely measures the time the bundle takes executing:

<script>_wml.perf.mark("before-bundle")</script>
<script src="https://ll-us-i5.wal.co/dfw/[hash]/v1/standard_js.bundle.[hash].js" id="bundleJs" defer></script>
<script>_wml.perf.mark("after-bundle")</script>

This code doesn’t work as expected: because of defer, the bundle executes after both of these inline scripts. Just in case somebody from Walmart is reading this.

Load non-important code only when necessary#

Chrome DevTools have the “Coverage” tab that analyzes how much CSS and JS is unused. If we open the tab, reload the page and click around a bit to run the most important JavaScript, we’ll see that around 40-60% of JS still hasn’t executed:

This code likely includes modals, popups and other components that aren’t rendered straight when the customer opens the page. They are a good candidate to be loaded only when actually needed. This might save us a few hundred kBs of JS.

This is how you load components dynamically with React and webpack:

import() docs

In a real app, you’ll likely want to use react-loadable instead

import React from 'react';

class FeedbackButton extends React.Component {
  handleButtonClick() {
    // ↓ Here, import() will make webpack split FeedbackModal
    // into a separate file
    // and download it only when import() is called
    import('../FeedbackModal/').then(module => {
      this.setState({ FeedbackModal: module.default });
    });
  }

  render() {
    const FeedbackModal = this.state.FeedbackModal;

    return <React.Fragment>
      <button onClick={this.handleButtonClick}>
        Provide feedback!
      </button>
      {FeedbackModal && <FeedbackModal />}
    </React.Fragment>;
  }
};

Don’t serve babel-polyfill in modern browsers#

If we look into standard_js.bundle.[hash].js, we’ll notice that it includes babel-polyfill:

Pretty easy to find by searching for “babel”

babel-polyfill weights 32.9 kB gzipped and takes 170 ms to download on Fast 3G:

By not shipping this polyfill in modern browsers, we could make the page fully interactive 170 ms earlier! And this is fairly easy to do:

  • either use an external service that serves polyfills based on User-Agent, like polyfill.io,
  • or build a second bundle without polyfills and serve it using <script type="module">, like in the Philip Walton’s article.

Don’t load polyfills multiple times#

Another problem is that the Object.assign polyfill is served in 3 files simultaneously:

The polyfill is small on its own, but this might be a sign that more modules are duplicated across the bundles. I’d try looking into that if I had access to sources.

Remove Node.js polyfills#

By default, webpack bundles polyfills for Node.js-specific functions when it sees them used. Theoretically, this is useful: if a library relies on setImmediate or Buffer which are only available in Node.js, it will still work in a browser thanks to the polyfill. In practice, however, I’ve seen the following happen:

// node_modules/random-library/index.js
const func = () => { ... };

if (typeof setImmediate !== 'undefined') {
  // ↑ Webpack decides that `setImmediate` is used
  // and adds the polyfill
  setImmediate(func);
} else {
  setTimeout(func, 0);
}

The library is adapted to work in the browser, but because webpack sees that it references setImmediate, it bundles the polyfill.

Node polyfills are small (a few kBs minified), so removing them usually doesn’t make sense. Still, it’s a good candidate to optimize if we were squeezing the last milliseconds from the page. Removing them is super easy (but needs to be tested – what if some code really needs them?):

// webpack.config.js
module.exports = {
  node: false,
};

Decrease the render-blocking CSS#

Apart from JS, page rendering is also blocked by CSS. The browser won’t render the page until all CSS (and JS) files are downloaded.

The Walmart page initially depends on two CSS files. In my tests, the largest of them takes even longer to download than the JS bundle – so it blocks rendering even after the script got downloaded and executed:

Notice how the page stays blank (look into “Frames” in the bottom half of the image) until the CSS is fully downloaded
Guardian Responsive Design, SmashingConf, 2013 (from 27:54)

How to solve this? We can go the way Guardian went in 2013:

  1. Find the critical CSS and extract it into a separate file. “Critical” means “The page looks funny without it”.

    Tools like Penthouse or Critical might be useful here. I’d also tune the result manually to exclude content that’s above the fold but is not very important (e.g., header navigation):

    We can show this a couple seconds later in exchange for faster overall rendering
  2. When serving the initial HTML, only load the critical CSS.
  3. Once the page is more or less downloaded (e.g., when the DOMContentLoaded event happens), dynamically add the remaining CSS:

    document.addEventListener('DOMContentLoaded', () => {
      const styles = ['https://i5.walmartimages.com/.../style.css', ...];
      styles.forEach((path) => {
        const link = document.createElement('link');
        link.rel = 'stylesheet';
        link.href = path;
        document.head.appendChild(link);
      });
    });
    

If we get this right, we’ll be able to render the page several hundred milliseconds earlier.

Remove duplicated styles#

In total, the Walmart page downloads three CSS files: one with the font definitions (BogleWeb.css) and two with the app styles (standard_css.style.[hash].css and style.[hash].css). The latter two seemed pretty similar, so I removed all the content except selectors and tried to compare the files.

Guess what? There’re 3400 common selectors among these files – and these selectors mostly have common styles! For the perspective, the first file has around 7900 selectors total, and the second one has around 4400:

The grep command is from StackOverflow

That’s a good area to optimize. This won’t affect time to first paint if we decrease the render-blocking CSS properly, but these CSS files will still load faster!

Add a service worker to cache assets#

The Walmart site is not a single-page app. This means that, on different pages, the customer has to download different styles and scripts. This makes every other page load longer, especially if the customer visits the site rarely.

We can improve that by creating a service worker. A service worker is a script that runs in the background even when the site is closed. It can make the app work offline, send notifications, and so on.

With Walmart, we can create a service worker that caches site resources in the background even before the user needs them. There’re multiple ways to do this; the concrete one depends on the Walmart infrastructure. A good example of one approach is available in the GoogleChrome repo.

Side note: notifications#

With service workers, we also get the ability to send notifications to customers! This should be used with caution – or we can annoy them – but this can increase engagement too. Good examples of notifications are “The product you saved for later got a discount” or “John Ford replied to your question about iPhone 8”.

To learn more, see the WebFundamentals’ guide into web push notifications.

Other ideas#

There’s still a room for further optimizations. Here’re some things that might also help – but we need to confirm that on the real app:

  • Using the local storage for caching large dependencies. The local storage seems to be several times faster than the HTTP cache. We might store large dependencies in the local storage to load them quicker:

  • Improving the time to first byte. Occasionally, the server spends too much time serving static resources. See the long green bars? That’s the time spent waiting for the server:

    These delays are non-deterministic – I’ve seen them pretty often during the analysis, but they keep happening with different resources every time – so this might be a network issue. Still, I’ve noticed them in WebPageTest results too.

  • Enabling Brotli compression. When you download a text resource from a server, the server would usually compress it with GZip and serve the compressed version. The browser will decompress it later, once received. This compression makes the text several times smaller.

    Apart from GZip, there’s also Brotli – a pretty new compression algorithm which compresses text 15-20% better. Right now, all text resources on the Walmart page are compressed with GZip. It makes sense to try Brotli to see if it improves the average download time.

Bonus Increase the product image quality#

That’s kinda related to performance too.

To reduce the size of the images, Walmart compresses them on the server side. The client specifies the image dimensions it expects to receive, and the server sends the appropriate image:

https://i5.walmartimages.com/[hash].jpeg?odnHeight=&odnWidth=&odnBg=

In most cases, this is great. However, for the primary product images, this has a negative effect. When buying an expensive gadget, I often make a final decision by visiting the product page to see the gadget, to imagine how it looks in my hands. But when I come to the Walmart site, I see a low-quality image with compression artifacts:

See yourself on WebArchive

I’d optimize this part for UX instead of performance – and serve images in a better quality. We can still keep the size difference minimal:

  • Try a different encoding algorithm. WebP is 30% smaller than JPEG given the same compression level. MozJPEG is an optimized JPEG encoder that works everywhere and has significantly less compression artifacts.
  • Use progressive images. Usually, during loading, images are rendered top-to-bottom: you see the top part of the image first, and then it fills
  • <picture> elementHTML5 Rocks tutorial

    Use the <picture> tag to stay compatible with different browsers. For example, we could serve WebP for Chrome and JPEG for other browsers:

    <picture>
      <source srcset="https://i5.walmartimages.com/[hash].webp?..." type="image/webp">
      <img src="https://i5.walmartimages.com/[hash].jpeg?...">
    </picture>
    
  • Serve Retina images with <source srcset>. Like this:

    <picture>
      <source
        srcset="https://i5.walmartimages.com/[hash].webp?odnHeight=450&odnWidth=450,
          https://i5.walmartimages.com/[hash].webp?odnHeight=900&odnWidth=900 2x"
        type="image/webp"
      >
      <img
        src="https://i5.walmartimages.com/[hash].jpeg?odnHeight=450&odnWidth=450"
        srcset="https://i5.walmartimages.com/[hash].jpeg?odnHeight=900&odnWidth=900 2x"
      >
    </picture>
    

Summing up#

So, to optimize the product page on the Walmart site, we can:

  • Fix the invisible text with font-display: optional
  • Use defer for the large JavaScript bundle
  • Load non-important code with webpack’s import
  • Remove polyfills in modern browsers
  • Decrease render-blocking CSS
  • Remove duplicated styles
  • Add a service worker for caching assets in background

With these tricks, we can render the product page earlier by at least 400-600 ms. If we apply similar improvements to the whole site, we can increase orders by at least 3–6% – and help Walmart earn more.

Thanks to Jacob Groß, Vladimir Starkov, and Anton Korzunov (in no particular order) for reviewing this post.


Backend for front-end devs: Part 1, Node.js

A few months ago, I had to build a full-stack app having only front-end experience. This is an overview of backend tools and practices if you discover yourself in the same situation.

I’m a front-end dev, but I need to write a full-stack app. What to do?#

To write an app, you’ll have to figure out three things: a programming language for the backend, a database to store data, and infrastructure.

Popular backend languages include Ruby, Python, JavaScript (Node.js), Java and PHP. If you’re a front-end developer, start with Node.js – things would be way easier.

I’ll cover databases and infrastructure in the next parts.

OK, I’ve just installed Node.js. What is it?#

Node.js enables you to write servers in JavaScript. It’s a JavaScript engine similar to the one Google Chrome has. You pass it a JS file, and the engine runs it:

// index.js
console.log('42');
# Shell
$ node ./index.js
42

To launch a real server, you’ll need to call a specific method provided by Node.js (e.g, createServer() from the net module). In practice, a few people do this – almost everyone uses high-level wrappers.

I’ve heard a bit about Node.js and callbacks. What’s the deal with them?#

Callback is a function that you pass into other function. It’s called when a specific action completes. MDN docs

The primary thing you need to know about Node.js is that it performs all long actions (like reading data from a database) asynchronously – using callbacks. This means that to make a DB request, you don’t do this:

const data = sql.query('SELECT name from users');
console.log(data);

but do this instead:

sql.query('SELECT name from users', (error, data) => {
  console.log(data);
});

Under the hood, Node.js sends a request to the database and immediately continues executing the code. And when the request completes, Node.js calls the callback and passes the data to it. This helps to write servers that handle lots of requests at the same time.

Can I write code without callbacks?#

Yes. If you don’t like passing functions into other functions, there’re a couple of alternatives:

  • Option A: use promises instead of callbacks. You’ll still have to pass functions around – but you’ll write code without excessive nesting which often happens with callbacks:

    Use util.promisify to adapt native callback-based APIs to promises
    // With callbacks
    const fs = require('fs');
    
    fs.readFile('./text.txt', (err, data) => {
      console.log(data);
    });
    
    // With promises
    const fs = require('fs');
    const util = require('util');
    const readFile = util.promisify(fs.readFile);
    
    readFile('./text.txt')
      .then((data) => {
        console.log(data);
      })
      .catch((err) => {
        // ...
      });
    
  • Option B: use async/await with promise APIs. This lets you write code that looks synchronous:

    async/await is available with Node.js 7.6+.
    // With callbacks
    const fs = require('fs');
    
    fs.readFile('./text.txt', (err, data) => {
      console.log(data);
    });
    
    // With async/await
    const fs = require('fs');
    const util = require('util');
    const readFile = util.promisify(fs.readFile);
    
    try {
      const data = await readFile('./text.txt')
      console.log(data);
    } catch (err) {
      // ...
    };
    

Many native Node.js APIs (like fs.readFile) also have synchronous alternatives (like fs.readFileSync):

const fs = require('fs');
const data = fs.readFileSync('./text.txt');

However, avoid using them on real servers – or the server would die just from a few simultaneous clients. (These APIs are useful in console scripts though.)

OK, how do I create a server?#

Use Express:

const express = require('express');
const app = express();

// ↓ Define a REST request
app.get('/api/user', (req, res) => {
  res.json({ name: 'Jason Bourne' });
});

// ↓ Ask the server to return statis files
// from the `public` dir (if the /api/user route didn’t match)
app.use('*', express.static('public'));

app.listen(3000, () => {
  console.log('Server is listening on port 3000')
});

Express is useful for building static servers and REST APIs.

How do I build something real?#

In a real app, apart from an HTTP server, you’ll need to implement a few more things. Here’re the common solutions for them:

  • Authorization: Passport.js. Passport.js works with login-password pairs, social networks, OAuth and lots of other login strategies. A SaaS solution like Auth0 is also an option
  • Validating requests: express-validator. express-validator normalizes and validates REST data you recieve from the clients
  • Sending emails: Nodemailer. Nodemailer works with SMTP; it also has simplified settings for AWS Simple Email Service
  • Password hashing: bcrypt. Important: learn how to store and hash passwords properly
  • Logging: loglevel, debug, or winston. I haven’t found a perfectly satisfying solution
  • Running the app in production: PM2. PM2 restarts an app in case of crashes, deploys a new version without a downtime, distributes the load to multiple instances and allows to deploy remotely

That’s it! The next part of the guide will introduce you into databases (specifically, into MySQL).

Designing APIs: use arrays instead of rest parameters

Or why _.get(object, ['deep', 'prop']) is better than _.get(object, 'deep', 'prop').

Imagine you’re designing a function that accepts a variable number of arguments. Like getDeepProperty():

function getDeepProperty(<object> and <chain of nested fields>) {
  // Retrieve the deep property and return it
}

And you’re facing a choice: should you pass the chain of fields directly:

function getDeepProperty(object, ...fields) { /* ... */ }

getDeepProperty(someData, 'i', 'love', 'designing', 'apis');

or as an array:

function getDeepProperty(object, fields) { /* ... */ }

getDeepProperty(someData, ['i', 'love', 'designing', 'apis']);

Earlier, I preferred the former way because it seemed “cleaner”. Now, I use the latter one – because it makes the function forward-compatible.

At some moment in the future, you might want to pass an additional parameter specifying the default return value – i.e., what the function should return if the deep property is absent. The only backwards-compatible way to do this is to pass this parameter as the last argument to the function, like this:

function getDeepProperty(
  <object>,
  <chain of fields>,
  <default return value>
) {
  // Retrieve the deep property and return it,
  // or return the default value
}

And here comes the difference. If your function accepts an array, you just append a new argument, do a switch (arguments.length) inside your function body, and the old code keeps working:

function getDeepProperty(...args) {
  if (args.length === 3) {
    const [object, fields, defaultValue] = args;
    // Execute the new behavior
  } else {
    const [object, fields] = args;
    // Execute the old behavior
  }
}

// The new API calls work great: a user passes three arguments,
// the function understands this and returns 'no'
getDeepProperty({}, ['i', 'love', 'designing', 'apis'], 'no');

// The old API calls keep working: a user passes two arguments,
// the function works as usual and e.g. throws an exception
getDeepProperty({}, ['i', 'love', 'designing', 'apis']);

But if your function accepts a variable number of arguments, you become unable to add an argument without breaking the old API users. The function can’t understand whether the last parameter is the default return value or a value in the chain of fields:

function getDeepProperty(object, ...fieldsAndDefaultValue) {
  // So, is fieldsAndDefaultValue[fieldsAndDefaultValue.length - 1]
  // a default value or just a field in a chain of fields?
  // You can’t know

  // Execute the new behavior
}

// The new API works great: the function returns 'no'
getDeepProperty({}, 'i', 'love', 'designing', 'apis', 'no');

// But the old API breaks: the function returns 'apis'
getDeepProperty({}, 'i', 'love', 'designing', 'apis');

So, here’s the rule:

When defining a function, prefer arrays over variable number of arguments

How to optimize images in webpack

Images take more than a half of the size of an average page:

A pie chart. Title: “Average bytes per page per content type.” Text in the bottom: “Total 3422 kB”. The largest pie chart section: “Images – 1818 kB.”

That’s a lot of traffic! But with webpack, it’s easy to decrease it.

1. Inline small PNG, JPG and GIF images#

Use url-loader to embed small PNG, JPG and GIF images into the bundle.

url-loader converts a file (if it’s smaller than the specified size) into a Base64 URL and inserts this URL into the bundle. This helps to avoid extra image requests (which is useful even with HTTP/2).

The limit of 5-10 KB is OK:

// webpack.config.js
module.exports = {
  module: {
    rules: [
      {
        test: /\.(jpe?g|png|gif)$/,
        loader: 'url-loader',
        options: {
          // Images larger than 10 KB won’t be inlined
          limit: 10 * 1024
        }
      }
    ]
  }
};

2. Inline small SVG images#

Use svg-url-loader to embed small SVG images.

This loader works like url-loader, but it encodes files using the URL encoding instead of the Base64 one. Because SVG is text, the result of the URL encoding is smaller.

The limit of 5-10 KB is also OK:

// webpack.config.js
module.exports = {
  module: {
    rules: [
      {
        test: /\.svg$/,
        loader: 'svg-url-loader',
        options: {
          // Images larger than 10 KB won’t be inlined
          limit: 10 * 1024,
          // Remove quotes around the encoded URL –
          // they’re rarely useful
          noquotes: true,
        }
      }
    ]
  }
};

3. Optimize image size#

Use image-webpack-loader to make images smaller.

This loader compresses PNG, JPG, GIF and SVG images by passing them through optimizers. Since it just pipes images through itself and doesn’t insert them into the bundle, it should be used with url-loader/svg-url-loader.

The default loader settings are OK:

// webpack.config.js
module.exports = {
  module: {
    rules: [
      {
        test: /\.(jpg|png|gif|svg)$/,
        loader: 'image-webpack-loader',
        // Specify enforce: 'pre' to apply the loader
        // before url-loader/svg-url-loader
        // and not duplicate it in rules with them
        enforce: 'pre'
      }
    ]
  }
};

Questions#

“Why is this important?”
As I’ve said, images take more than 50% of the average page size, and optimizing them is super-easy. Through, at the same time, I’ve rarely seen webpack configs that do it. We seriously should always optimize them.

“What should I do?”
Go and add these loaders to your config.

“Are the any side effects?”
A couple:

  • image-webpack-loader increases the build time, so it’s better to disable it during development (pass the bypassOnDebug: true option to do that)
  • url-loader and svg-url-loader remove extra image requests, but break caching of these images and increase the JS loading/parsing time and memory consumption. If you inline large images or lots of images, you might experience issues. (Thanks to Addy Osmani for noting this.)

Further reading#

How to effectively ask for permissions in web

Matt Wilcox writes that it’s meaningless to ask a person to subscribe for notifications when they’re visiting a site for the first time.

He’s right. Here’s how to ask for permission to make it effective:

1. Find the right moment#

In mobile apps, 60-70% of users dismiss the permission request if it’s requested straight on the first launch.

Asking the user to subscribe for notifications when they visit a site for the first time is ineffective. They don’t know anything about you yet. It’s likely they’ll just block your request.

Instead, find the right moment:

Wrong Right Why
Asking to share the location when a user opens a pizzeria site Asking to share the location when a user clicks “Show pizzerias nearby” on a pizzeria site In the latter case, it’s clear why the site is asking for the permission. In the former case, it’s not
Suggesting to subscribe for notifications when a user visits a news site for the first time Suggesting to subscribe for notifications when a user visits a news site for a third time in a week In the latter case, the user is frequent, so they might find notifications relevant. In the former case, they don’t even know what content the site has

2. Show a custom permission request at first#

This is how e.g. a Russian media called TJournal does this:

A custom non-modal popup in the top left corner of the window. Text: “Would you like to subscribe to important news from TJ?” Buttons: “Yes, I would” and “Close”
“Would you like to subscribe to important news from TJ?” – “Yes, I would”

TJournal won’t trigger the browser request until the user clicks the “Yes, I would” button. There’re two reasons for this:

  • If the user blocks the browser request, you won’t be able to ask for the permission again. On the contrary, if the user dismisses your custom request, you’ll still be able to ask again in a different situation – e.g., when a user clicks on a “Subscribe” button in a different part of site.
  • The browser request is non-customizable. You can’t put a picture or a custom text there. But you can with a custom request.

3. Finally, if a user accepted the request, show them the browser dialog#

This one:

That’s it.

Further reading#

Describe your React propTypes as deeply as possible

On my project, I’ve been noticing code like this:

class Header extends React.Component {
  static propTypes = {
    items: PropTypes.array,
    counters: PropTypes.object,
  };
  
  // ...
}

This is how it should look like instead:

class Header extends React.Component {
  static propTypes = {
    items: PropTypes.arrayOf(PropTypes.shape({
      id: PropTypes.string.isRequired,
      name: PropTypes.string.isRequired,
      link: PropTypes.string.isRequired,
    })).isRequired,
    counters: PropTypes.objectOf(PropTypes.number).isRequired,
  };

  // ...
}

The difference is that in the latter component, propTypes are much more detailed. It’s better for two reasons:

  • React validates your props better. In the former component, you won’t get any warnings if you pass a wrong array into items or if you forget to pass it at all. Instead, you’ll have a wrong rendering result or a runtime error and will have to debug it.
  • You understand the interface of the component easier. This is even more important.

    With the latter component, to understand the structure of items, you just look at its propTypes. With the former one, you have to dive into its code. It’s not a problem when the component has been created just 10 minutes before, and you remember what it accepts, but it makes further support way easier.

There’s only one case when I find it acceptable to skip some propTypes definitions. It’s when your component just passes a prop to a child component and doesn’t care about its structure. In this case, the child component should validate the prop:

Note how the items
propType in Header cares only about the id field it uses, and counters doesn’t care about its items type at all.
class Header extends React.Component {
  static propTypes = {
    items: PropTypes.arrayOf(PropTypes.shape({
      id: PropTypes.string.isRequired,
    })).isRequired,
    counters: PropTypes.objectOf(PropTypes.any).isRequired,
  };

  render() {
    return <div>
      {this.props.items.map(item =>
        <HeaderItem item={item} counter={this.props.counters[item.id]} />
      }
    </div>;
  }
}

class HeaderItem extends React.Component {
  static propTypes = {
    item: PropTypes.shape({
      name: PropTypes.string.isRequired,
      link: PropTypes.string.isRequired,
    }).isRequired,
    counter: PropTypes.number.isRequired,
  };

  // ...
}

Here’s the rule:

Describe your propTypes as deeply as possible

Short basics of caching

I’ve finally configured caching on my site. Here’s what you need to know.

Headers#

There’re 4 headers that enable caching: Cache-Control/Expires and Last-Modified/ETag. The former two are primary; they enable caching and instruct the browser on how long it should save a resource. The latter two are secondary and optional; browsers use them when the cached resource gets expired to check if it has changed. (If it hasn’t, browsers just take the expired one and keeps using it.)

In practice, you choose 2 of these headers: one primary and one secondary – and use them together. Using all four headers isn’t very practical – the browser will rely on only two of them.

I recommend choosing Cache-Control and ETag. Cache-Control lets you configure the caching details that Expires can’t. ETag, according to MDN, is more reliable than Last-Modified.

Use Cache-Control and ETag

Lifecycle#

Imagine you have a file called pic.gif. This one:

This is how its lifecycle will look:

  1. The browser requests pic.gif. The server sends the response with, for example, these caching headers:

    Cache-Control: max-age=60 
    ETag: deadbeef123
    
  2. The user refreshes the page. If less than 60 seconds have passed (60 is a value from Cache-Control: max-age), the browser doesn’t make any requests and just takes pic.gif from the cache.
  3. The user refreshes the page. If more than 60 seconds have passed (60 is a value from Cache-Control: max-age), the browser sends a request for pic.gif and attaches the If-None-Match: deadbeef123 header. (deadbeef123 is a value from the ETag header that the browser has received.)

    When the server receives the request, it reads pic.gif and calculates its ETag (ETag is a hash which changes when the file changes). And then:

    • if the calculated ETag is still deadbeef123, the file hasn’t changed. The server returns an empty response with the 304 Not Modified status.
    • if the calculated ETag is different, the file has changed. The server returns the content of pic.gif with the 200 OK status and new Cache-Control and ETag headers.
Servers respond with either 304 Not Modified or with a new resource

Total immutability#

Usually, on the third step (↑), when the resource expires, the browser makes a request to the server. If the file has changed, the browser downloads the new version. But if it hasn’t, the browser receives 304 Not Modified and doesn’t download anything. That’s nice, but the extra request to the server is still there.

Each network request brings a delay

If you know that the resource will never change, and checking this doesn’t make any sense, you can send the immutable value in the Cache-Control header:

Cache-Control: max-age=60, immutable

or simply

Cache-Control: immutable

After this, the browser will always consider the cached resource as valid and will never send a verification request for it. It’s like if you set max-age to 1000 years.

Cache-Control: immutable is a new header value. At the moment, it works only in Firefox and Edge.

Use Cache-Control: immutable to prevent any additional requests

Versioning#

On my site (except this blog), I enabled Cache-Control: immutable for all images, styles, and scripts. To force the browser to re-download the file if it changes, I append the last change date to the file name:

/images/pic.gif?hash=1499433448

This way, the browser will send a request for the file only when the file (and, therefore, its name) changes.

This approach—including a dynamic value into the file name—is called versioning. Versioning is a common practice, and I recommend enabling it in your app.

Unlike versioning, Cache-Control: immutable hasn’t become a common practice yet. However, it’s already being used by e.g. Facebook, so you can try enabling it too.

Use versioning, it’s a common practice

Summing up#

  • Use Cache-Control and ETag headers for caching
  • Implement versioning for cache invalidation
  • Try Cache-Control: immutable if you have resources that will never change

Malicious packages in npm. Here’s what to do

Here’s all the information I’ve found.

💻☠️🏴

What happened?#

People found malicious packages in npm that work like real ones, are named similarly real ones, but collect and send your process environment to a third-party server when you install them:

This is dangerous because, on CI servers, the environment usually includes different secret tokens.

What to do if I’m a user?#

Regenerate the secret tokens if you installed any package from these as a dependency:

A screenshot of the cached page with packages

npm has also confirmed this list

babelcli - v1.0.1 - Babel CLI for Nodejs
crossenv - v6.1.1 - Run scripts that set and use environment variables across platforms
cross-env.js - v5.0.1
d3.js - v1.0.1 - d3.js for Nodejs
fabric-js - v1.7.18 - Object model for HTML5 canvas, and SVG-to-canvas parser. Backed by jsdom and node-canvas.
ffmepg - v0.0.1 - FFmpeg for Nodejs
gruntcli - v1.0.1 - Grunt CLI for Nodejs
http-proxy.js - v0.11.3 - Node.js proxy tools
jquery.js - v3.2.2-pre - jquery.js for Nodejs
mariadb - v2.13.0 - A node.js driver for mysql. It is written in JavaScript, does not require compiling, and is 100% MIT licensed.
mongose - v4.11.3 - Mongoose MongoDB ODM
mssql.js - v4.0.5 - Microsoft SQL Server client for Node.js.
mssql-node - v4.0.5 - Microsoft SQL Server client for Node.js.
mysqljs - v2.13.0 - A node.js driver for mysql. It is written in JavaScript, does not require compiling, and is 100% MIT licensed.
nodecaffe - v0.0.1 - caffe for Nodejs
nodefabric - v1.7.18 - Object model for HTML5 canvas, and SVG-to-canvas parser. Backed by jsdom and node-canvas.
node-fabric - v1.7.18 - Object model for HTML5 canvas, and SVG-to-canvas parser. Backed by jsdom and node-canvas.
nodeffmpeg - v0.0.1 - FFmpeg for Nodejs
nodemailer-js - v4.0.1 - Easy as cake e-mail sending from your Node.js applications
nodemailer.js - v4.0.1 - Easy as cake e-mail sending from your Node.js applications
nodemssql - v4.0.5 - Microsoft SQL Server client for Node.js.
node-opencv - v1.0.1 - OpenCV for Nodejs
node-opensl - v1.0.1 - OpenSSL for Nodejs
node-openssl - v1.0.1 - OpenSSL for Nodejs
noderequest - v2.81.0 - Simplified HTTP request client.
nodesass - v4.5.3 - Wrapper around libsass
nodesqlite - v2.8.1 - SQLite client for Node.js applications with SQL-based migrations API
node-sqlite - v2.8.1 - SQLite client for Node.js applications with SQL-based migrations API
node-tkinter - v1.0.1 - Tkinter for Nodejs
opencv.js - v1.0.1 - OpenCV for Nodejs
openssl.js - v1.0.1 - OpenSSL for Nodejs
proxy.js - v0.11.3 - Node.js proxy tools
shadowsock - v2.0.1 - A tunnel proxy that help you get through firewalls
smb - v1.5.1 - A Pure JavaScript SMB Server Implementation
sqlite.js - v2.8.1 - SQLite client for Node.js applications with SQL-based migrations API
sqliter - v2.8.1 - SQLite client for Node.js applications with SQL-based migrations API
sqlserver - v4.0.5 - Microsoft SQL Server client for Node.js.
tkinter - v1.0.1 - Tkinter for Nodejs

Here’s also a one-liner that will list these packages if any of them was installed as a dependency:

Twilio also listed one-liners for checking the whole file system at once (for bash and Powershell)
npm ls | grep -E "babelcli|crossenv|cross-env.js|d3.js|fabric-js|ffmepg|gruntcli|http-proxy.js|jquery.js|mariadb|mongose|mssql.js|mssql-node|mysqljs|nodecaffe|nodefabric|node-fabric|nodeffmpeg|nodemailer-js|nodemailer.js|nodemssql|node-opencv|node-opensl|node-openssl|noderequest|nodesass|nodesqlite|node-sqlite|node-tkinter|opencv.js|openssl.js|proxy.js|shadowsock|smb|sqlite.js|sqliter|sqlserver|tkinter"

Always check the name of packages you’re installing. You can look at the downloads number: if a package is popular but the downloads number is low, something is wrong.

What to do if I’m a library developer?#

I see two options:

  • Use scopes (@scope/package-name) for your packages. With scopes, it’s harder to install a wrong package accidentally: a user would have to misspell both the scope name and the package name. Unfortunately, it’s not: it’s enough to misspell just the scope name (e.g @babel/babel-cli@bable/babel-cli). Scopes might help a bit because they can have a simple name that’s harder to misspell, but it’s still not a universal solution.
  • Take the most common misspellings of your packages by yourself. Think of the most common misspellings and publish empty packages under these names. You can also warn users about the right name with npm deprecate.

Is this even OK?#

This isn’t surprising – npm doesn’t have any protection against this yet. In fact, that’s why there could be other malicious packages. Stay careful and check package names.

npm is working on a solution though:

You can also participate in the Yarn’s discussion about a white list for preinstall/postinstall script packages.


How to optimize resizing or scrolling

If you listen for events like mousewheel, resize or scroll, your page could get slow. These events are generated multiple times per second, and if the event handler takes too much time, the browser won’t catch with redrawing the page.

This is what to do if you have such problem.

Throttle#

By default, the event handler is executed on each event. Sometimes, when the handler is taking a long time, it might be acceptable to delay its execution to once per 100-200 ms. This “delaying” can be achieved with the _.throttle utility method.

When is this useful? E.g. if you recalculate the page layout each time the browser resizes. If recalculating layout takes a long time, you could perform it less often.

Before and after:

// Before
window.addEventListener('resize', () => {
  calculateLayout(); // Takes 20 ms
});

// After
window.addEventListener('resize', _.throttle(() => {
  calculateLayout(); // Still takes 20 ms, but now runs once in 100 ms
}, 100));

passive: true#

passive: true a flag that’s passed into addEventListener. It tells the browser that the event handler will never call event.preventDefault(). Using this knowledge, the browser can start doing the event action (e.g. scrolling) without waiting for the JS code to finish. This makes the page smoother.

Currently, only touchstart, touchmove and mousewheel event handlers could be made passive.

When is this useful? E.g. if you redraw the parallax background on each touch or scroll event. Most likely, it won’t be a problem if the browser starts scrolling before the background finishes redrawing. If so, such event handler could be made passive.

Before and after:

// Before:
window.addEventListener('touchstart', updateBackground)

// After:
window.addEventListener('touchstart', updateBackground, {
  passive: true
})

Use alternative APIs#

For some common tasks, there are alternative APIs that help to perform the task easier and with better performance:

  • Intersection Observer. It helps if you need to detect when an element goes off the screen or intersects with another element. This API can replace the scroll and resize listeners.
  • window.matchMedia. This API can notify you when a specific media query becomes true. Helps if you need to e.g. update the page layout when the viewport reaches a specific width. It can replace the resize listeners.
  • position: sticky. This CSS property lets you stick an element to the top when it reaches an edge of the viewport. Helps if you need to e.g. make a sticky page header. It can replace the scroll listeners that switch the element from position: static to position: fixed and vice versa.
  • At the moment of July 2017, the API is not yet well-supported. However, there’s a polyfill.

    Resize Observer. It helps if you need to detect when an element gets resized. This API can replace the scroll and resize listeners.

    Thanks to Vitali Kuzmich for mentioning this approach.

Optimize the code itself#

You can also make the event handler itself run faster. It should take not more than 10 ms to run.

To analyze how much time your handler takes, use Web Performance API or the Performance tab in Chrome DevTools:

The optimization approach depends on your code. Try caching some data or skipping some computations. If you use React, you can try dropping parts of the render tree with shouldComponentUpdate.

And this is what won’t work:#

  • requestAnimationFrame. It doesn’t help with slow event handlers and, by my tests, doesn’t bring any benefits for fast ones. Seems like it’s only useful for scheduling JS animations.
  • Event delegation. It optimizes the memory usage, not the speed.

That’s it. Optimize.

Case study: improving a popular library’s size for webpack users

There’s a library called Polished. It’s a utility collection for writing styles in JavaScript.

The polished logo

And it had a problem.

Problem#

A story in three tweets:

So, this code:

import { opacify, transparentize } from 'polished'; 

generates a much larger bundle than this code:

import opacify from 'polished/lib/color/opacify.js';
import transparentize from 'polished/lib/color/transparentize.js';

even despite the Polished’s bundle is built with ES modules and tree-shaking is enabled.

Let’s find out what causes this.

Investigation#

1. Verify the entry point#

Environment: polished@1.2.0 and webpack@2.6.1 (webpack@3.0.0 gives the same result)

At first, let’s check that import { ... } from 'polished' picks up a file written with ES exports. If it doesn’t, webpack can’t do any tree-shaking at all.

When you import a package, webpack understands what exact file to use by looking into specific fields in package.json. Polished’s package.json has two of them:

{
  "name": "polished",
  "description": "A lightweight toolset for writing styles in Javascript.",
  "main": "lib/index.js",  // This one
  "module": "dist/polished.es.js",  // And this one
  ...
}

Webpack prefers module over main. module points to dist/polished.es.js, and this file does have an ES export:

// polished/dist/polished.es.js
...
export { adjustHue$1 as adjustHue, ... };

This point is OK.

2. Check if there’s unused code that’s unnecessarily kept#

polished/dist/polished.es.js is written with ES exports. This means that tree-shaking should work properly, and the unused imports shouldn’t be included into the bundle. Then why different imports produce different file sizes?

Side effect is when a function changes something outside of itself – e.g. writes a value to a global variable or initiates a network request

The most possible reason is that polished/dist/polished.es.js contains some code that’s absent in our polished/lib/... files and that can’t be simply dropped by the tree-shaker. This is the code that could cause side-effects. E.g. if a file includes a top-level function call, the tree-shaker can’t remove the function even if its result isn’t used. The function could be causing side effects, and removing it could break the app.

Let’s compare the bundles that we have after importing polished in two different ways and verify this case.

To do this, I create a package:

# Shell
mkdir polished-test && cd polished-test
npm init -y
npm install polished webpack@2

Add two files that import Polished in two different ways:

console.log() helps finding the index.js file in the bundle + prevents webpack from removing the imports as unused
// index-import-package.js
import { opacify, transparentize } from 'polished';

console.log('polished', opacify, transparentize);

// index-import-files.js
import opacify from 'polished/lib/color/opacify.js';
import transparentize from 'polished/lib/color/transparentize.js';

console.log('polished', opacify, transparentize);

Add a special webpack configuration that emits two bundles:

// webpack.config.js
const webpack = require('webpack');

module.exports = {
  entry: {
    // We’ll compare two different bundles,
    // thus two different entry points
    'bundle-import-package': './index-import-package.js',
    'bundle-import-files': './index-import-files.js',
  },
  output: {
    filename: '[name].js',
    path: __dirname,
  },
  plugins: [
    // We need to run UglifyJS to remove the dead code
    // (this will do tree-shaking), but prevent it
    // from uglifying the code (so it’s easier to read the bundle)
    new webpack.optimize.UglifyJsPlugin({
      // Disable several optimizations so that the bundle
      // is easier to read
      compress: { sequences: false, properties: false, conditionals: false, comparisons: false, evaluate: false, booleans: false, loops: false, hoist_funs: false, hoist_vars: false, if_return: false, join_vars: false, cascade: false },

      // Beautify the bundle after uglifying it
      beautify: true,

      // Don’t rename the variables
      mangle: false,
    }),
  ]
}

And run the build:

./node_modules/.bin/webpack

Now, I have two bundles, each with a different approach to importing stuff. I open them in my editor and switch to the structure view to their content. And here’s what I see:

A comparison between the content of two files. The left file is bundle-import-package.js, it has a lot of functions. The right file is bundle-import-files.js, it has much less functions.

bundle-import-package.js has more methods than bundle-import-files.js. Most likely, they are kept because of calls with side effects. Let’s dig deeper.

3. Find the exact cause of the problem#

So, bundle-import-package.js has a lot of functions that aren’t used but are still included. If we look through the file to see their usages, we’ll see a large snippet of code like this:

// bundle-import-package.js
// ...
function opacify(amount, color) {
    // ...
}
var opacify$1 = curry(opacify);
function desaturate(amount, color) {
    // ...
}
curry(desaturate);
function lighten(amount, color) {
    // ...
}
curry(lighten);
// ...

Here, desaturate and lighten are those unused functions, and opacify is a function we import in the client code.

This code comes to bundle-import-package.js from polished/dist/polished.es.js. The corresponding code in that file looks like this:

// polished/dist/polished.es.js
// ...
function opacify(amount, color) {
    // ...
}

var opacify$1 = curry(opacify);

function desaturate(amount, color) {
    // ...
}

var desaturate$1 = curry(desaturate);

function lighten(amount, color) {
    // ...
}

var lighten$1 = curry(lighten);
// ...

And this code comes into polished/dist/polished.es.js from the library sources. This is how it looks:

// polished/src/color/opacify.js
function opacify(amount: number, color: string): string {
  // ...
}

export default curry(opacify);

// polished/src/color/desaturate.js
function desaturate(amount: number, color: string): string {
  // ...
}

export default curry(desaturate);

// polished/src/color/lighten.js
function lighten(amount: number, color: string): string {
  // ...
}

export default curry(lighten);

So what happens here? dist/polished.es.js is built with Rollup. When the library authors do a build, Rollup grabs all the modules and converts exports (export default curry(lighten)) into variable assignments (var lighten$1 = curry(lighten)).

When we do import { opacify, transparentize } from 'polished', webpack tries to compile dist/polished.es.js and drop the unused code. It removes the desaturate$1 and lighten$1 variables because they aren’t exported, but it can’t drop the curry(darken) calls because curry could produce side-effects. And because functions like desaturate and lighten are passed into curry(), they are also kept in the bundle.

Screenshot of the editor
This is how you analyze the bundle: open the file structure, find a function that’s absent in the other bundle, and search for its usages

Solution#

To decrease the bundle size, we should do one of the following things:

Pure function is a function that doesn’t produce side effects
  • tell UglifyJS that it’s safe to remove curry() calls because it’s pure
  • or move currying into the functions instead of wrapping them.
Another option is passing compressor: { pure_funcs: ['curry'] } to the UglifyJS options, but Polished can’t control this

To tell UglifyJS that curry() calls are safe to remove, we have to mark each call with the /*#__PURE__*/ annotation. This way, the minifier will understand that this call is pure and will be able to optimize it:

We can’t just add the /*#__PURE__*/ annotation after export default. Rollup seems to remove comments if they are placed in that position
// polished/src/color/lighten.js
function lighten(amount: number, color: string): string {
  // ...
}
 
- export default curry(lighten);
+ const curriedLighten = /*#__PURE__*/curry(lighten);
+ export default curriedLighten;

The second approach is to move currying into the functions body. With it, we should do something like this:

// polished/src/color/lighten.js
- function lighten(amount: number, color: string): string {
-   // method body
- }
+ function lighten(...args) {
+   return applyCurried(function (amount: number, color: string): string {
+     // method body
+   }, args);
+ }
 
- export default curry(lighten);
+ export default lighten;

I prefer the first approach because it (almost) doesn’t complicate the code.

After adding the /*#__PURE__*/ annotations, minified bundle-import-package.js goes from 16 down to 11.8 kB. But that’s not the end – bundle-import-files.js is still smaller (9.86 kB). This is because there’re a few other places that should be optimized.

I’ll skip the part where I find them and jump right to the solution.

  • Change 1 and 2. Like with curry(), there’re two other places where the export is wrapped into a function. It’s polished/src/helpers/em.js and polished/src/helpers/rem.js. To optimize them, we should similarly add the /*#__PURE__*/ annotations.
  • Change 3. In polished/src/mixins/normalize.js, there’re two global objects that use computed object properties. When they are compiled, Babel transforms them to call the Babel’s defineProperty function. Because of this, UglifyJS can’t remove them. To solve the problem, we should either move these objects into the normalize() function that uses them or wrap them into getter functions.

And, when we apply these additional optimizations, we’ll have this:

                   Asset     Size  Chunks             Chunk Names
  bundle-import-files.js  9.87 kB       0  [emitted]  bundle-import-files
bundle-import-package.js  7.76 kB       1  [emitted]  bundle-import-package

bundle-import-package.js is now even smaller than bundle-import-files.js! Great.

I’ve submitted the pull request.