This is true. Also equivalent to "Chalta hain" is "Jugaad". We really need to stop doing this. These short cuts/hacks are okay for when you don't have the means/resources available. Not when you have abundance. Doesn't make any sense whatsoever!
It's very hard to switch the mindset though. It becomes ingrained such that even when abundance is the new state of affairs scarcity still dominates the mind.
Can confirm. I grew up in India and now live in the US. I regularly regress into a scarcity-oriented mindset (e.g. preventing all food wastage, avoiding spending on "nice-but-not-strictly-necessary" things, etc.).
Even though this mindset is incongruous and inefficient for my present circumstances -- and I'm aware of it -- overcoming childhood programming about it is difficult.
The one DB fits all approach only works when the size of the database is really small and never grows. Imagine you have 100 customers. Each customer generates, on average, a million 1536 dimension vector embeddings (considering OpenAI Ada dimensions which is the most popular right now). That is 6GB (1536 x 4 bytes per dimension for f32 x 1000_000) of just embeddings PER CUSTOMER. If you use HNSW it will take at least that much of RAM if not more. If you use PQ (and variants) you can reduce the size of index in RAM to say 512MB-1GB per customer. It is still quite a lot of memory requirement. That is just the way it is and there is no way around it.
Now imagine you are using that database for storing transactions and other day to day business ops that will still be storing millions of records but with small indexes. This would have ideally only required a single DB instance with a replica for redundancy. Now if you integrate Vectors into the equation, you will have to needlessly scale this DB both horizontally and vertically just to maintain a decent query/write performance to your DB (which would have ideally been extremely fast without embeddings in the mix). You will eventually separate the embeddings out as it makes no sense for the entire DB to be scaled just for the sake of scaling your embeddings. I am not even accounting for index generation for these vectors which will require nearly 100% of all CPU cores while the index is being generated (depending on type of ANN you are using) and which in turn would slow your DB to a crawl.
Exactly - vector indexes are so different than traditional RDBMS B-Tree or LSM Tree indexes that it doesn’t make sense to use the same store for both unless it’s basically a toy app.
Someone makes the example in another comment, but it’s analogous to OLTP vs OLAP
My experience is that if you do the data modelling properly a well designed star schema with some aggregation tables or materialized views on top can often remove the need for dedicated OLAP software.
Now you do NOT want to run such a setup on the same hardware that you use for your transactional systems, of course. But you CAN use the same software (like Oracle), which means that you do get some reduction in tech complexity.
Are there any DB that could support both use cases while being able to partition them in such a way that the transactions etc are only kept on part of the resources they need to be. Basically two seperated DBs but sharing the same interfaces and security etc.
What you are talking about is possible to do in regular SQL dbs with extensions. However, when it comes to scaling traditional DBs don't have the necessary tools to do so automatically. Most extensions provide support for an underlying ANN algorithm it implements and there's that and nothing more. Everything else you'll have to hand roll yourself.
Clustering, load balancing, aggregating queries etc are quite different for a vector database in comparison to traditional OLTP databases.
It's the same as difference between OLAP vs OLTP. Both have different underlying architectural differences which make it incompatible for both to run in an integrated fashion.
For instance, in a traditional DB the index is maintained and rebuilt alongside data storage and for scaling you can separate it into read/write nodes. The write nodes typically only focus on building indexes while the read nodes for querying eventually consistent indexes (eventual consistency is achieved by broadcasting only the changed rows rather than sending entire index).
Now it's similar in vector dbs too. You can seperate the indexer from query nodes (which access eventually consistent index). However, the load is way higher than a regular db as the index is humongous/takes a long time to build and sharing the index with query nodes is also more time consuming and resource/network intensive, as you won't be sharing few rows but the entire index itself. It requires a totally different strategy to get all query nodes to be eventually consistent.
The only advantage of traditional DBs also implementing vector extensions is familiarity for the end user. If you are already familiar with postgres you wouldn't want to leave your comfort zone. However, scaling a traditional DB is different from scaling a vector DB and you'll encounter those pain points only in production and will be forced to switch to proper vector databases anyways.
We (PlanetScale) announced Vector storage and search today. If I am understanding your request it sounds like something could do. I would love to hear more if you are willing to chat? s@planetscale.com
Absolutely this is par for the course for distributed engines.. just not postgres and other single node engines which a lot of people here will tell you are all you need.. what they mean is that's all you need until you either have to introduce enormously complex application tier sharding or you're moving to a scalable engine
Read all your replies. I can only conclude that have no understanding of Hindus, Sikhs or even India. Forget understanding Modi. That is not your cup of tea. I see a lot of comments on HN mindlessly bash Modi based on the ridiculous, perverted view presented by Western Media. That is what is driving your conclusions. Indians know better as to what exactly is happening in India and why we voted Modi. The West will never understand either the political landscape of India or the dynamics of demographic politics. For starters, majority of the "oppression" BS in India (be it on religious, caste or gender lines) is manufactured by all parties (representing all strata of society) to garner sympathy and votes from gullible voters. The actual reality is quite the opposite. You cannot have a prosperous economy, in a Democratic setup, with continuous oppression. Just because few riots happened in some parts of the Country, does not mean the Government is authoritarian. Do not forget that India is a country of 1.4 Billion People. It would be ridiculous to generalize the Nation, its Elected Government or the People based on extreme issues (which are bound to happen considering how vast the Country is and how huge the population is).
Either ways, as sibling comments have rightfully noted, you are unable to grasp the true psyche of India and Indians. For you to truly understand India, you will have to stop referring to Western publications of India (which are so out of touch with ground reality it is hilarious) and move to India and live her for at least a decade. You will reach the same conclusions as Indians then.
> 50$ will make our investors very happy but then customers might complain.
Look at the pain points of customers. It is obvious they are willing to pay higher price for convenience. Anyone visiting India would not complain about $50. Especially since you say it is a one time fee for opening the account. You need to rethink your price. You can even make it an annual fee so you have consistent revenue. The current pricing model is nuts. $10 is too cheap for foreigners.
EDIT: You write elsewhere that there is a markup of 3.5% on top per transaction. Is this credit card fees or your own markup on top of the credit card fees? If it is the latter, then I would advise you against it. Rather take 1 time payment annually ($50 or even $100) instead of 3.5% markup per transaction.
Even if you put a flat $30-$50 fee, make it have limited validity (with an expiration date). Like you have in SIMs. Many foreigners buy throwaway SIMs on arrival. Most of these SIMs have limited validity. You can use the same concept here as well. It makes no sense to charge a flat fee for eternity. Also, coming to think of it, tying up with a SIM provider would be a great idea. In the same kiosk/booth, foreigners can purchase SIM as well as get a Cheq verified account. You can share a % revenue with SIM provider and not have to hire extra staff. The SIM provider would have all the necessary setup anyways for physical verification. Just reuse that setup.
I cracked up at the little flag waving by who I assume was the prime minister/president. The duality is interesting serious Indian scientists doing great work and a politician on the screen for no good reason. Here’s to the tribe of scientist prevailing and taking the country forward, prob true everywhere now that I think of it.
> The duality is interesting serious Indian scientists doing great work and a politician on the screen for no good reason
One can say the same about President Nixon talking to the Astronauts who landed on the Moon. At least here ISRO comes directly under the Prime Minister of India while NASA does not come under the President of USA.
The Prime Minister is the head of Department of Space, whose primary arm is ISRO. He is the boss. Literally every Prime Minister of India has attended ISRO launches and addressed ISRO after success/failure of launches.
> asked Modi to "bless us"
You probably aren't Indian but this is quite common in India where we seek blessings from elders.
That might have been an issue with the control room getting the timing of the live feed wrong. No reason to read into it more than that. We have had previous missions where the Prime Minister's live feed was beamed after the event concluded
No, he's just an elected representative of the people, a government servant.
> You probably aren't an Indian
I am an Indian.
Modi, desperate to project himself, is overruling the President (who is the supreme commander of the armed forces), gotten himself "coronated" in the new Parliament building, gotten his picture put on every citizen's COVID vaccination certificate (as if he has invented the vaccine).
PM is not the boss of the executive according to the constitution*. The fact that traditionally and more recently the President follows the whim of the PM does not negate the text that identifies the elected President as the head of the executive.
> identifies the elected President as the head of the executive.
Wrong. He is a "nominal head", or in other words, "in name only". The powers are vested in the Prime Minister not the President. So it does not make him the head of the executive. Read Article 53 (3)a of Indian Constitution which overrides 53(1) and resolves any ambiguity: https://indiankanoon.org/doc/1597349/
(3) Nothing in this article shall
(a) be deemed to transfer to the President any functions conferred by any existing law on the Government of any State or other authority;
So the authority of Prime Minister, as defined by the Constitution, cannot be transferred to the President, even though he is a nominal head. This is as per 53 (3)a.
"(1) There shall be a Council of Ministers with the Prime Minister at the head to aid and advise the President who shall, in the exercise of his functions, act in accordance with such advice: Provided that the President may require the council of Ministers to reconsider such advice, either generally or otherwise, and the President shall act in accordance with the advice tendered after such reconsideration"
Which means the President can only ask for advice (from PM + Council of Ministers) to be reconsidered, and after reconsideration, he still has to act on the advice. He cannot overrule it. Hence why a Bill can be passed by Parliament and sent back by President to be reconsidered only once. The second time the Bill is sent, the President is duty bound to assent else he can be impeached. The Prime Minister on the other hand cannot be impeached. The ruling Government can be brought down with a no-confidence motion or if enough members from ruling party defect. No other way to impeach a Prime Minister.
Also read Article 77:
77. (1) All executive action of the Government of
India shall be expressed to be taken in the name of the
President.
It is pretty clear that he has no "real" executive powers and is head in "name" only as the Government is taking actions in his name. The executive power rests with Government of India whose head is the Prime Minister.
It is also even more undermined by the fact that the President is not directly elected by the people (like in US where people vote for the President and representative votes are cast through their electoral college) but through an electoral college consisting of the elected representatives of the People. He has no real executive powers vested in him unlike the President of USA who can pass executive orders.
Tell me one executive order passed by President of India directly and unilaterally. There are none.
> No, he's just an elected representative of the people, a government servant.
And he still is the boss of DoS whose arm is ISRO. You can cry all you want, it won't change facts.
> Modi, desperate to project himself, is overruling the President (who is the supreme commander of the armed forces), gotten himself "coronated" in the new Parliament building, gotten his picture put on every citizen's COVID vaccination certificate (as if he has invented the vaccine).
ISRO is primary arm of the Department of Space which is headed by the Prime Minister. So in essence, the Prime Minister is the boss. It is not an independent federal agency like NASA.
I really wish to see a lander mission to Venus. Doesn't look like anybody other than Russia has done it - that too nearly 40 years ago. The environment is so extreme that the technology - especially electronics - would have to be radically different. The data is also likely to be extremely interesting.
Have you considered training a shallow MLP autoencoder, perhaps with tied weights between the encoder and the decoder to reduce the dimensionality of the embeddings? Another (IMO, better) approach I can think of off the top of my head, would be to use a semi-supervised contrastive learning approach, with labelled similar and dissimilar video pairs, like in this notebook[1] from OpenAI.
We have built a few video-search system by now, using USearch and UForm for embedding. They are only 256 dims and you can concatenate a few from different parts of the video. Any chance it would help?
If you ever want to understand what living in a tech bubble is like, reading the comments here should be sufficient. People feeding off each others conspiracies and hate for Musk. Hilarious to watch
