A private key in the context of Bitcoin is a secret number that allows bitcoins to be spent. Every Bitcoin wallet contains one or more private keys, which are saved in the wallet file. The private keys are mathematically related to all Bitcoin addresses generated for the wallet.
Because the private key is the "ticket" that allows someone to spend bitcoins, it is important that these are kept secure. Private keys can be kept on computer files, but in some cases are also short enough that they can be printed on paper.
In Bitcoin, a private key is a 256-bit number, which can be represented in one of several ways. Here is a private key in hexadecimal - 256 bits in hexadecimal is 32 bytes, or 64 characters in the range 0-9 or A-F.
E9873D79C6D87DC0FB6A5778633389F4453213303DA61F20BD67FC233AA33262
Some wallets allow private keys to be imported without generating any transactions while other wallets or services require that the private key is swept. When a private key is swept, a transaction is broadcast that sends the balance controlled by the private key to a new address in the wallet. Just as with any other transaction, there is risk of swept transactions to be double-spending.
In contrast, bitcoin provides a facility to import a private key without creating a sweep transaction. This is considered very dangerous, and not intended to be used even by power users or experts except in very specific cases. Bitcoins can be easily stolen at any time, from a wallet which has imported an untrusted or otherwise insecure private key - this can include private keys generated offline and never seen by someone else.
A Bitcoin address is a 160-bit hash of the public portion of a public/private ECDSA keypair. Using public-key cryptography, you can "sign" data with your private key and anyone who knows your public key can verify that the signature is valid.
A new key pair is generated for each receiving address (with newer HD wallets, this is done deterministically). The public key and their associated private keys (or the seed needed to generate them) are stored in the wallet data file. This is the only file users should need to backup. A "send" transaction to a specific Bitcoin address requires that the corresponding wallet knows the private key implementing it. This has the implication that if you create an address and receive coins to that address, then restore the wallet from an earlier backup, before the address was generated, then the coins received with that address are lost; this is not an issue for HD wallets where all addresses are generated from a single seed. Addresses are added to an address key pool prior to being used for receiving coins. If you lose your wallet entirely, all of your coins are lost and can never be recovered.
Bitcoin allows you to create as many addresses as you want, and use a new one for every transaction. There is no "master address": the "Your Bitcoin address" area in some wallet UIs has no special importance. It's only there for your convenience, and it should change automatically when used.