Dealing with huge data in Vue.js

Hi Yavor,

I am trying to load a huge file with data into my single-page app. It’s 6 MB and the app takes 3 minutes only to load the file.

Later I use the file to create a select box with a list of clients in it. Each line includes a Client name + City.

Here is a screenshot:

Screenshot

Labels are in Dutch, but I think it’s all rather self-explanatory ….

I have two major problems:

  • Loading the data takes a lot of time
  • The app slows down very much when the data is loaded and scrolling/searching the list of clients is pretty much impossible.

Greetings,
Johnny

Hey Johnny!

I see what your problem is. The problem as a whole is that you are trying to load too much data at once and this is making the browser choke – first at loading the data and then at operating it. So what I would suggest is solutions that reduce the amount of data loaded.

Solution One

Get rid of the scrolling list/box as a whole. Why? I have seen select boxes with several hundred entries (Select Font boxes in Photoshop) but not more than this. More than a few hundred entries makes a select pretty much unusable, even on desktop. Even in DB clients… No DB client loads the whole DB, they load 100 records at most…
Google is dealing with a ton of records but they never use a select box. They do however have an auto-suggest box:

Google search

 So I would go for google’s solution.

How to implement it

  1. Make an endpoint on your API. Something like /clients/search/(string).
  2. Make an onKeyUp event listener on your search box.
  3. When the user has no “keyUp” for 0.5 seconds (use something like this – “debounce” it )- send an AJAX request to /clients/search/(input here) and fetch the first 5 results you find in the database. Display them as selectable options.
  4. That’s pretty much it.

Solution two

If you really really REALLY insist on the (useless in my opinion) select box. You might try to implement some kind of Infinite Scroll. Open the link and keep scrolling down… And keep scrolling. It’s loading 10 results at first and when you keep scrolling it loads more through AJAX.

Solution Three

Have a simple paginated list/table of users. Load only 50 of them per page. When the user chooses another page – just load them through AJAX (and remove the previous page from memory).

Solution Four

Mix One + Three: Give them the search bar, also give them the paginated list.. but still for 150K records, using a paginated list might not be the best

Solution Five

  1. Give them a list of Alphabet letters Something like
    First Name: A B C D E F G …
    Last  Name: A B C D E F G …
    City: A B C D E F G …
  2. When they click a letter load a paginated list of users
  3. Speaking of pagination… You might want your API to return something like

And you would like to maybe send it something like:

Just to be sure what you are “purchasing” to be delivered from it.


But again. Do NOT load all the data at once. This will keep things snappy.

Hope it helps!

  • Johnny Driesen

    Hi Yavor,

    Thanks for creating this blogpost !
    Just a small remark …
    It takes me 3 minutes (and not 3 secs) to load all 150K records.
    If it would be 3 secs, I could perfectly live with it 🙂 😉

    FYI: It seems there’s also 1 picture missing (just below the paragraph ‘Google is dealing’ …

    As written in my mail. rather quickly, I found out that loading the 150K records ‘in one shot’ is certainly not the way to go…

    And you’re right … The scrollbar probably is never going to be used in this particular situation …
    Though I’m having other ‘screens’ (views) in my App, with less records, and I want to use the same ‘component’…
    So I’m searching for a re-usable solution…

    I don’t think pagination is a nice solution (pure aesthetically speaking….)
    IMO pagination is the perfect solution for ‘tables’, but not really for a list as shown in my mockup…

    Solution 5 doesn’t seem to be that practical…
    The user has to be able to search on ‘part of strings’ as well…

    I think I’m going for a combination of solution 1 and solution 2.
    I knew already the ‘infinite list’ solution (in Angular) you mentioned.
    Translating it into VueJS can’t be that hard 🙂

    Yavor, THANKS again for your effort and time
    As soon as I’m having a working solution, I’ll let you know and I’ll provide you the sources.

    Kind regards,
    Johnny

    • developermode

      You are welcome, Johnny!

      I am happy to hear you find the two first solutions to be most appropriate. I think your end users would benefit from them as well.

      Have a great weekend! 🙂

  • Johnny Driesen

    Hi Yavor,

    FYI: Feel free to make the mockup a bit bigger…
    There are not secrets in it 🙂 🙂

    Grtz,
    Johnny

  • Juan Lamberto N

    Nice blog!!! I hope to learn ViewJS asap

    • Thank you Juan! Will post more.