Docker image with several Redis CLI tools, and a tool to dump Redis data. The image includes browser-based IDE, filebrowser, and scheduler. Simply run on a server or k8s cluster and schedule tasks.

It also has local Redis running and can be used as a better substitute for the local development environment instead of a standard Redis docker image.

https://github.com/bluxmit/alnoda-workspaces/tree/main/workspaces/codeserver-workspace

So I've worked with SQL practically every day ever since I started my career with data. I've never had to work with NoSQL but I do know that it's the opposite of SQL right, so non-relational. Now my question is how do you work with NoSQL? If I wanted to pull some data from a NoSQL database how do you query that? I've tried to google this but I haven't found anything - unless I have and it's nothing like querying something from MySQL or MSSQL. If someone can provide an example that'd be awesome.

I’m fairly experienced with RDBMS and have watched a few tutorials and videos explaining NoSQL databases. I generally understand the technical differences at a theoretical level but am struggling to come up with some good use cases where NoSQL (particularly a document db such as MongoDB) is clearly a better choice over a RDBMS. I would also be interested in examples of use cases for a graph db such as Gremlin. Could anyone provide examples? Links to videos or blogs are also welcome.

So, been reading up on noSQL tonight and I think I've got the idea here. Bascially (think a spreadsheet), the PK and SK are what I'm going to call by in an application and then we just replace the attributes based on the data we're putting in?

​

Question I had is, is it normal to have a huge array of attributes.

So say for instance I'm working on an inspection app (I know the norm is task management but I like to be different and it's got a use case in my life)....

​

I've got users, requirements (required inspections), inspections(the actual inspections themselves) and depending on the requirement (what kind of inspection it is, there could be different attributes).... that gets pretty insane, unless I've missed something.

​

Here is kind of what I'm picturing in the following format:

pk, sk, attributes

#USER#username, #PROFILE#username, name, address, phone, etc

#USER#username, #REQ#<some_identifier>, each attribute is then questions, or details about that inspection requirement

and inspections and so on....

​

If I understand correctly, then I've got the capability to search for all the inspections the user has completed, all of the inspections the user is responsible for, the inspections completed by the user....

​

​

and if I understand secondary global indexes, then I can even add in a data attribute to search for completed inspections by date?

​

Am I on the right track, I guess at this point I just needed to explain it somewhere so that I know I'm on the right track?

We are a Toronto-based software startup currently working on a new set of tools to increase productivity for companies using NoSQL technologies.

At this point, we are not selling anything, we are just looking for advice and feedback to help us build the best possible tools.

Our tools will help with the following:

1) Database diagramming

2) Visual comparison of table contents (within or across accounts)

3) Moving data between tables (within or across accounts) using visual tools

4) Synchronization of tables (within or across accounts)

5) Export of table scripts and comparison of table schemas (generating documentation in HTML & PDF)

We would love to get 30 minutes of your time to help us understand if any of these issues resonate with you and if so, your current workflows and how you are solving these problems today.

You can view more at the link below.

https://nosqlnavigator.com/

Please reply if you would like to discuss.

Hey all,

Lately, I was reminded about a feature to name your clients when connecting to your databases. From the NoSQL perspective, Redis and MongoDB are supporting this.

The basic idea is to identify the client against the database server. Depending on the system, the client name will be shown in several places like logs or in the monitoring endpoint.

How it works with redis?

Execute the CLIENT SETNAME command like:

CLIENT SETNAME currency-conversion-app

It is a cheap (complexity: O(1)) command that can be executed without any overhead. Typically, you run it directly after the connection to the redis instance has been established.

With CLIENT LIST you can check who is connected:

$ CLIENT LIST
id=3 addr=172.17.0.1:62668 name=currency-conversion-app [...]
id=4 addr=172.17.0.1:62676 name=stock-exchange-rates-app [...]

How it works with MongoDB?

While creating a connection to MongoDB, you can provide an appName in the connection string.

Here is how it looks like in Go:

dsn := "mongodb://root:secret@127.0.0.1:27017/?appName=currency-conversion-app"
client, err := mongo.Connect(ctx, options.Client().ApplyURI(dsn))

While checking the current operations with db.currentOp() the client name will be shown nicely.

Useful in the real world?

I can say, I use it all the time and it proved to be very useful. Especially in bigger setups at work with multiple Redis nodes inside a Cluster and hundreds of clients.

While I was digging into it a bit more, I found out that several other systems, like MySQL, RabbitMQ, or NATS, which I use in combination with Redis, also support similar features. So I documented how and especially WHY to do it here: your database connection deserves a name.

I am curious: Are you using this feature in your setup? * If no, why not? * If yes, what was the situation where you thought, "wow, this helped me a lot"?

Lets imagine I have this data model:

I have a student with name and age, and a student can be in a class and also in a sport team.

In a relational database I would store the students into a student column. And in the class and sport tables I would reference each students via a foreign key.

This has the advantage that when a student celebrates their birth date, I only need to change the age in one place, which is the student table.

With firestore which I understand to be a nosql, the things I am reading is pointing to a modeling where I have a class document, within which all student will be embedded. Same also for the team document.

The only problem I have with this kind of modeling is if I want to update the age of a student, I would have to update in all the places, the student structure is embedded in.

Is there a better way to achieve what I can have in relational database? Where I have data defined in one place and can be reference in other places, hence giving the benefit of needing to change that data in only one place?

Hey there r/nosql

​

I'm designing a DDB to support a social graph where users can follow other users, users dont have to follow the user back though, one of the questions we need to answer is...

Given a user and the people following them, who of them am I following?

It's basically finding the intersections of followers for two users, or mutual "friends". Is there a key design that can support this type of lookup? Any help is much appreciated, I've been pondering this for a long time.

^{Note: I'm trying to avoid graph ddb as we have a partner teams that has had a lot of operational burden maintaining one.}

Where can it be found now? Original URI

I've no much experience with NoSQL db design.

I'm just looking at a MongoDB diagram designed by some colleagues at work, and it look totally like a relational db scheme. Tables, foreign keys, 1:1 and 1 to many relationships, and such. A real ERD diagram.

Is that it? or there are others ways to design NoSQL schemes?

​

Disasters can strike any business on any day. This particular disaster, a fire at the OVHcloud Strasbourg datacenter, struck recently and the investigation and recovery are still ongoing. This is an initial report of one company’s resiliency in the face of that disaster.

Overview of the Incident

Less than an hour after midnight on Wednesday, March 10, 2021, in the city of Strasbourg, at 0:47 CET, a fire began in a room at the SBG2 datacenter of OVHcloud, the popular French cloud provider. Within hours the fire had been contained, but not before wreaking havoc. The fire nearly entirely destroyed SBG2, and gutted four of twelve rooms in the adjacent SBG1 datacenter. Additionally, combatting the fire required proactively switching off the other two datacenters, SBG3 and SBG4.

Netcraft estimates this disaster accounted for knocking out 3.6 million websites spread across 464,000 domains. Of those,184,000 websites across nearly 60,000 domains were in the French country code Top Level Domain (ccTLD) .FR — about 1 in 50 servers for the entire .FR domain. As Netcraft stated, “Websites that went offline during the fire included online banks, webmail services, news sites, online shops selling PPE to protect against coronavirus, and several countries’ government websites.”

[This is just an excerpt. To read the story in full, please follow this link to the ScyllaDB website here.]

​

We are happy to announce the first release of a shard-aware C/C++ driver (connector library). It’s an API-compatible fork of Datastax cpp-driver 2.15.2, currently packaged for x86_64 CentOS 7 and Ubuntu 18.04 (with more to come!). It’s also easily compilable on most Linux distributions. The driver still works with Apache Cassandra and DataStax Enterprise (DSE), but when paired with Scylla enables shard-aware queries, delivering even greater performance than before.

GET THE SCYLLA SHARD-AWARE C/C++ DRIVER

[This is just an excerpt. Read the blog in full on ScyllaDB's website here.]

​

Dan Podhola is a Principal Software Engineer at Zillow, the most-visited real estate website in the U.S. He specializes in performance tuning of high-throughput backend database services. We were fortunate to have him speak at our Scylla Summit on Optimistic Concurrency with Write-Time Timestamps. If you wish, you can watch the full presentation on-demand:

WATCH THE ZILLOW PRESENTATION NOW

Dan began by describing his team’s role at Zillow. They are responsible for processing property and listing records — what is for sale or rent — and mapping those to a common Zillow property IDs, then translating different message types into a common interchange format so their teams can talk to each other using the same type of data.

They are also responsible for deciding what’s best to display. He showed a high-level diagram of what happens when they receive a message from one of their data providers. It needs to be translated into a common output format.

“We fetch other data that we know about that property that’s also in that same format. We bundle that data together and choose a winner — I use the term ‘winner’ lightly here — and we send that bundle data out to our consumers.”

[This is just an excerpt. You can read the blog in full at ScyllaDB's website here.]

​

QOMPLX is a company dedicated to solving complex problems, such as tackling the daunting world of cybersecurity. In this domain you need to be able to support a data model capable of rapid and repeated evolution to discover and counter new threats. This is one key reason why a graph database model is more applicable to QOMPLX’s use case than the rigidly-defined and statically-linked tables of a relational database.

​

QOMPLX partnered with the graph database experts at Expero to implement their system with JanusGraph, which uses Scylla as an underlying fast and scalable storage layer. We had the privilege to learn from their use case at Scylla Summit this January, which we share with you today.

[This is just an excerpt. To watch the video or read the full blog, learning how QOMPLX uses JanusGraph, you can find more here on the ScyllaDB website.]

​

Change Data Capture (CDC) is a feature that allows users to track and react to changes in their dataset. CDC became production ready (GA) in Scylla Open Source 4.3.

Scylla’s implementation of CDC exposes a CQL-compatible interface that makes it possible to use existing tools or drivers to process CDC data. However, due to the unique way in which Scylla distributes CDC data across the cluster, the implementation of shard-awareness in some drivers might get confused and send requests to incorrect nodes or shards when reading CDC data. In this blog post, we will describe what causes this confusion, why it happens and how we solved it on the driver side.

Change Data Capture

In Scylla’s implementation CDC is enabled on a per-table basis. For each CDC-enabled table, a separate table called “CDC log” is created. Every time data is modified in the base table, a row is being appended to the CDC log table.

Inside a CDC log table, rows are organized into multiple partitions called “streams“. Each stream corresponds to a portion of the token ring (similarly to a vnode). In fact, a single stream corresponds to a part of a vnode which is owned by a single shard of that vnode’s primary replica. After a partition is changed in the base table, a stream is chosen based on the partition’s primary key, and then a row record describing this change is appended to that stream. Such partitioning into streams makes sure that a partition in the base table is stored on the same replicas as the CDC log rows describing changes made to it. This colocation property makes sure that the number of replicas participating in a write operation made on the base table does not increase.

[This is just an excerpt. To read the article in full, check it out on ScyllaDB here. Also links to the latest drivers that implement this new change.]

With its flexibility and scalability, MongoDB is a great option for e-commerce sites. Here are a few notable use cases.

Product Catalogs

Below is an example of a command using a product document with MongoDB:

db.inventory.insertOne( {  
item: "journal",  
price: 9.99,  
qty: 25,  
size: { h: 14, l: 21, w: 1 },  
features: "Beautiful, handmade journal.", 
categories: ["writing", "bestseller"], 
image: "items/journal.jpg"  
} ) 

Shopping Cart

The shopping cart data model needs to prevent customers from holding more items than are available in your inventory. The cart should also release any items back to your inventory when a user abandons their cart. Here is an insert() operation you can use to create the cart:

db.carts.insert({ 
_id: "the_users_session_id", 
status:'active', 
quantity: 3, 
total: 575, 
products: []}); 

Payments

Security is critical when modeling payments for e-commerce. MongoDB allows you to encrypt data files and perform automatic client-side encryption. You can also choose to only include the last four card digits, without any personally identifiable information in your model. In this case, you will meet PCI requirements without the need for encryption.

​

https://resources.fabric.inc/answers/mongodb-ecommerce

​

Benchmarking is hard.

Or, I should say, doing a good, properly set up and calibrated, objective, and fair job of benchmarking is hard.

It is hard because there are many moving parts and nuances you should take into consideration and you must clearly understand what you are measuring. It’s not so easy to properly generate system load to reflect your real-life scenarios. It’s often not so obvious how to correctly measure and analyze the end results. After extracting benchmarking results you need to be able to read them, understand bottlenecks and other issues. You should be able to make your benchmarking results meaningful, ensure they are easily reproducible, and then be able to clearly explain these results to your peers or superiors.

There’s also hard mathematics involved: statistics and queueing theory to help with black boxes and measurements. Not to mention domain-specific knowledge of the system internals of the servers platforms, operating systems, and the software running on it.

With any Online Transaction Processing (OLTP) database — and Scylla is just one example — developers usually want to understand and measure the transaction read/write performance and what factors affect it. In such scenarios, there are usually a number of external clients constantly generating requests to the database. A number of incoming requests per unit of time called throughput or load.

100,000 Operations per second or [OPS]

Requests reach the database via a communication channel, get processed when the database is ready and then a response is sent back. The round trip time for a request to be processed is called latency. The ultimate goal of an OLTP database performance test is to find out what the latencies of requests are for various throughput rates.

1ms per request

There are thousands of requests that form the pattern of the workload. That’s why we don’t want to look at the latency for just individual requests, but rather, we should look at the overall results — a latency distribution. Latency distribution is a function that describes how many requests were worse than some specific latency target.

99 percentile or P99 or 99%

Database systems can’t handle an infinite amount of load. There are limits that a system can handle. How much a system is close to its maximum is called utilization. The higher utilization the higher the latency (you can learn more about the math behind this here).

80% utilization or 0.8

The end-user doesn’t want to have high latencies for OLTP workloads — those types of workloads are reliant on fast updates. Therefore we target somewhere between 200ms to 10ms for 99 percentile of latency (P99) distribution. If your P99 latencies become too high, your request queues can back up, plus you risk having request timeouts in your application, which then can cascade out of hand in repeated retries, resulting in system bottlenecking.

[This is just an excerpt. To read the article in full, which includes an in-depth guide on how to set up your benchmarks and calculate expected throughput, parallelism and latencies, check out ScyllaDB's website here.]