Your question is impossible to answer since there are too many parameters that can affect it.

For example to start with we need to define what the word "degrading" means to you?
What is acceptable for your application?

But another important question is how many columns and how large they will be in your table, this affects the size of the database which greatly affects the answer?

And what size of hardware will you be running on?

General advice like this is impossible to answer, a query time can differ 1000 times or more depending on the factors above.

But I'm going to try with a very general and rough advice though, which is that as long as the database is small enough to fit in RAM (including indexes) on your database server and you have indexes that can be utilized by the queries your queries should return under a second even with rows in the millions without a too fancy hardware.

But depending on what your queries look like they could well end up 100-1000 times slower so don't take my word for granted.